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CHAPTER  1 


INTRODUCTION 


Problem  Statement 

Estimating  and  budgeting  for  risk  and  uncertainty  in 
the  Air  Force  and  the  Department  of  Defense  (DOD)  is  hap¬ 
hazard  and  inconsistent.  A  DOD  Acquisition  Steering  Group 
report,  dated  March  31,  1981,  stated  that  "Early  cost,  sched 
ule  and  performance  estimates  are  consistently  overly  opti¬ 
mistic  and  highly  unrealistic  [23:8]."  Because  of  this, 
project  funds  requested  may  not  be  adequate.  For  management 
purposes : 

The  Military  Departments  should  provide  a  reserve 
for  unavoidable  contingencies  in  RDT&E  [Research, 
Development,  Test  &  Evaluation]  and  initial  production 
and  support  of  weapons  and  material  systems.  The  con¬ 
tingency  reserve  should  be  based  on  the  program  manager' 
assessment  of  program  risks  and  fully  supported  in  an 
approved  acquisition  strategy  [23:12]. 

If  we  in  DOD  expect  to  respond  to  this  call  for  management 

of  risk  and  uncertainty  costs,  certain  questions  need  to  be 

answered.  These  are:  How  are  risk  and  uncertainty  treated 

throughout  DOD?  What  effects  do  these  various  treatments 

have  on  our  ability  to  estimate,  budget,  and  manage?  What 

are  the  advantages  and  disadvantages  of  each  approach?  Does 

any  one  approach  appear  to  be  superior? 


Definition  of  Terms 

In  order  to  better  understand  what  is  meant  by  risk 
and  uncertainty,  the  following  example  is  presented.  Sup¬ 
pose  that  the  family  car  is  beginning  to  show  the  result  of 
age  and  high  mileage,  but  we  anticipate  a  year  of  service 
before  any  major  repair.  Let  us  assume  we  do  riot  wish  to  pay 
the  high  interest  in  making  payments  for  a  car,  so  we  wish 
to  put  enough  money  aside  to  cover  the  cost  of  purchasing  a 
new  replacement.  The  choice  of  how  much  money  to  put  aside 
is  complicated  by:  (1)  What  do  we  want?  What  kind  of  per¬ 
formance  or  capability  is  desired?  (2)  What  kinds  of  price 
changes  do  we  expect  in  the  intervening  period?  (3)  Will  our 
car  really  last  out  the  year? 

How  does  this  situation  demonstrate  risk,  uncertainty, 
and  the  use  of  management  reserves?  We  would  face  uncer¬ 
tainty  in  deciding  that'  our  car  will  make  it  one  year  since 
we  have  only  a  subjective  feel  for  the  probability  of  a 
breakdown  occurring  based  on  past  experience.  We  also  face 
uncertainty  in  that  our  requirements,  for  performance  might 
change  in  a  year. 

We  would  feel  risk  concerning  the  future  of  a  new 
car.  We  do  not  know  exactly  what  it  will  cost  because  costs 
change  slightly  from  year  to  year,  but  we  could  develop  an 
objective  distribution  of  expected  cost. 

The  only  event  that  is  certain  is  the  present  listed 
price  on  the  new  car  that  meets  our  current  needs.  This  is 
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certain  because  we  have  a  copy  of  the  written  quote  from  our 
favorite  dealer  which  includes  all  the  options  on  the  new 
car. 

We  must  figure  out  how  much  to  set  aside  for  the  new 
car  to  be  purchased  one  year  hence.  The  dealer-quoted  price 
becomes  the  basis  of  our  estimate.  By  adding  on  an  amount 
for  anticipated  price  increase  we  now  have  an  approximate 
amount  "to  put  aside  as  savings.  But  this  is  not  all.  The 
old  car  could  break  down  before  the  year  is  up.  In  addition, 
we  could  change  our  minds  about  the  performance  required. 

If  so,  we  would  have  to  choose  a  different  new  model  at  a 
different  price.  The  extra  money  added  to  cover  these  con¬ 
tingencies  is  termed  reserve  since  it  is  being  set  aside  to 
cover  unknown  future  events.  It  is  a  management  reserve 
since,  as  the  family  financial  manager,  we  have  placed  this 
extra  in  the  family  savings  based  upon  future  contingencies 
that  are  uncontrollable. 

The  following  terms  will  be  used  throughout  this 
report  and  are  defined  as  follows: 

1.  Risk  -  an  event  or  situation  characterized  by  an 
uncontrollable  random  variable  or  event  which  comes  from  a 
known  objective  probability  distribution  (27:82). 

2.  Uncertainty  -  an  event  or  situation  characterized 
by  an  uncontrollable  random  variable  or  event  which  comes 
from  an  unknown  subjective  probability  distribution  (27:82). 


3 


3.  Management  reserve  -  money  set  aside  by  a  man¬ 
ager  to  cover  uncontrollable  changes  or  events  (6:112). 

Certainty  -  an  event  or  situation  characterized 
by  complete  or  total  knowledge;  a  controllable  situation. 

5.  Cost  overrun  -  the  difference  between  expected 
cost  when  a  contract  was  signed  and  the  current  expected 
cost  (17:1-1  to  1-7). 

6.  Cost  growth  -  the  difference  between  the  first 
estimate  and  the  current  estimate  of  costs  (17:1-1  to  1-7). 
Although  the  terms  "risk"  and  "uncertainty"  are  used  inter¬ 
changeably  in  much  of  the  literature,  it  is  necessary  to 
understand  that  they  are  theoretically  different  concepts. 
Both  contribute  to  cost  overruns  and  cost  growth. 

Background 

The  problem  of  how  to  estimate  costs  and  manage  them 
for  a  given  project  is  not  new.  Shortly  before  the  beginning 
of  the  Christian  era,  the  Romans  decided  to  build  an  aque¬ 
duct  for  the  Asian  town  of  Troas .  From  the  very  beginning  of 
construction,  costs  were  exceeding  the  estimates.  As  con¬ 
struction  proceeded,  costs  were  more  than  double  the  esti¬ 
mate.  'Thanks  to  the  generosity  of  Julius  Atticus,  a  wealthy 
resident  all  of  the  extra  costs  were  paid  so  that  the 
project  could  be  completed  (22:2). 

Early  U.  S.  Naval  ship  construction  had  similar 
problems.  The  construction  of  six  large  frigates  was 


authorized  by  Congress  in  179^ •  By  the  end  of  1795,  the 
number  of  ships  to  be  purchased  was  cut  to  three  due  to 
delays  and  cost  overruns  (33:116).  In  this  case,  Congress 
limited  the  expansion  of  the  U.  S.  Navy  to  the  funds  origi¬ 
nally  allocated. 

Presently  DOD  is  faced  with  the  same  problems  of 
how  to  accurately  estimate  costs  and  control  cost  growth. 
Funds  are  limited  and  DOD  has  to  compete  with  other  increas¬ 
ing  needs  of  the  nation  such  as  social  welfare  programs, 
energy,  etc.  Since  the  taxpayers  are  not  like  the  wealthy 
Julius  Atticus,  cost  overruns  bring  great  public  outcry.  In 
fact , 

The  rampant  growth  in  the  acquisition  cost  of  most 
defense  systems  has  been  a  great  concern  to  the  defense 
community  for  the  past  several  years.  From  the  pro¬ 
nouncements  on  Capitol  Kill  to  the  reports  in  the  news 
media,  criticism  is  focused  on  the  Department  of  Defense 
for  its  rapid  expansion  of  cost  on  major  programs 
[  25:931- 

In  one  instance,  Senator  Sam  Nunn  indicated  the  U.  S.  is 
headed  toward  "unintentional,  gradual,  unilaterial  disarma¬ 
ment  by  cost  overrun  [12:^-B]."  President  Reagan,  in  an 
unpublished  presidential  memorandum  to  the  Secretary  of 
Defense  (17  March  1981),  "requested  that  the  Secretary  of 
Defense  assess  the  situation  and  initiate  actions  to  reduce 
cost  growth  [  25:93]."  Apparently  this  had  already  begun  in 
DOD  since  the  Under  Secretary  of  Defense,  William  A.  Long, 
discussed  cost  growth  and  other  acquisition  problems.  In 
his  memorandum,  dated  23  December  1981,  he  stated: 


r 
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Despite  some  initial  steps  [discussed  in  an  earlier 
memorandum  dated  30  April  1331],  controlling  cost  growth 
(both  real  and  perceived)  remains  a  major  problem.  The 
solution  must  include  more  realistic  estimates  accurately 
reflecting  future  costs  and  difficult  choices  to  reduce 
requirements  when  costs  grow  [24:4]. 

On  31  March  1981,  a  DOD  steering  group  report  pre¬ 
sented  the  results  of  "a  30-day  DOD  effort  to  identify  and 
assess  options  for  major  improvements  in  the  acquisition  pro¬ 
cess  [23:11]."  A  copy  of  this  report  was  sent  to  the  Deputy 
Secretary  of  Defense,  Frank  C.  Carluccl.  Two  recommendations 
of  particular  interest  are  Budget  to  Most  Likely  Cost  and 
Incorporate  the  Use  of  Management  Reserve  (23:18-19)-  These 
indicate  that  a  significant  portion  of  cost  growth  cou^d  be 
attributed  to  our  methods  for  estimating  and  managing  funds 
for  risk  and  uncertainty. 

On  30  April  1981,  Mr.  Carlucci  issued  the  thirty-one 
actions  known  as  the  Carlucci  initiatives.  Two  of  the  ini¬ 
tiatives  addressed  budgeting  problems.  They  were  Budget  to 
Most  Likely  Cost  (Number  Six)  and  Budget  Funds  for  Technologi 
cal  Risk  (Number  Eleven) (24 : 6-1, 11-1) .  These  two  required 
the  Services  to  "Budget  to  most  likely  cost  or  expected  costs 
Including  predictable  cost  increases  due  to  risk ,  instead  of 
the  contractually  agreed-upon  cost  [23:6-1],"  and  "budget 
funds  for  risk .  In  particular,  each  service  should  review 
the  TRACE  concept  and  either  adopt  it  or  propose  an  alterna¬ 
tive  for  its  use.  .  .[24:11-1]."  Neither  of  these  initia¬ 

tives  addressed  uncertainty.  In  discussing  Program  Stability 
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(Number  Pour),  cost  and  schedule  uncertainty  are  mentioned 
(23':^-2).  Overall  though,  the  document  containing  the 
Carlucci  initiatives  did  not  relate  risk  and  uncertainty  in 
any  specific  way.  It  would  appear  that  the  term  risk  is 
used  to  also  convey  the  idea  of  uncertainty . 

The  subject  of  setting  aside  money  to  cover  risk  and 
uncertainty  is  a  very  sensitive  political  issue.  Any  good 
manager  realizes  that  the  early  phases  of  research  and  devel 
. opment  (R&D)  are  characterized  by  high  risk,  since  the  con¬ 
cepts  are  still  being  defined  and  many  times  there  has  been 
no  forerunner  on  which  to  base  an  accurate  cost  estimate. 
While  all  three  services  would  admit  to  looking  at  risk 
analysis  and  uncertainty,  they  remain  somewhat  silent  on  the 
use  of  management  reserves  which  would  be  necessary  to  prop¬ 
erly  implement  these  into  a  budget.  Although  there  is  a 
high  level  of  interest  in  the  analysis  of  risk  and  uncer¬ 
tainty  and  the  creation  of  a  visible  management  reserve  to 
cope  with  factors  that  contribute  to  cost  overrun,  the  per¬ 
ception  that  "...  establishment  of  a  management  reserve 
is  contrary  to  prescribed  policy  for  budgeting  and  obligat¬ 
ing  funds  [6:112]"  remains. 

Despite  this  perceived  policy,  the  program  manager 
is  faced  with  unknown  changes  in  his  program  brought  about 
by  either  technological  advances,  changing  systems  require¬ 
ments,  changing  mission  requirements,  unexpected  cuts  by 
Congress  (i.e.,  decreases  or  deletions  in  appropriations  for 
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DOD  requirements),  etc.  In  order  to  cope  with  these  situa¬ 
tions,  the  program  manager  has  to  build  in  extra  money  to 
cover  these  uncertainties.  These  monies  are  termed  "manage¬ 
ment  reserves"  unofficially  since  they  represent  an  Informal 
way  of  funding  for  risks  and  uncertainty. 

Research  Objectives 

The  research  objective  is  to  better  identify  the 
issues  concerning  the  use  of  management  reserves  to  budget 
for  uncertainty.  The  subobjectives  are  to: 

-  review  management  reserve  practices  In  DOD 

-  identify  and  evaluate  techniques  used  to  estimate 

costs 

-  Identify  and  evaluate  techniques  used  to  estimate 
resource  requirements  for  risk  and  uncertainty 

-  evaluate  the  strengths  and  weaknesses  of  the  best 
technique (s) . 

Scope 

This  effort  was  limited  to  Research  and  Development 
(R&D)  practices  since  this  is  an  area  where  risk  and  uncer¬ 
tainty  are  the  highest  and  has  been  the  primary  concern  of 
past  DOD  and  service  efforts.  Historically,  the  conceptual 
phase  and  the  full-scale  development  phase  have  been  the 
most  difficult  areas  in  which  to  conduct  a  risk  analysis. 

The  production  phase  has  been  easier  to  model  as  the  program 
is  better  defined. 
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It  is  well  known  by  cost  analysts  that  the  management 
and  control  of  risk  and  uncertainty  involves  the  proper  selec¬ 
tion  of  technology,  materials,  performance  characteristics, 
schedule  determination,  and  proper  production  planning. 
Presently  the  services  have  been  tasked  to  improve  risk  man¬ 
agement  via  the  use  of  management  reserves.  Therefore,  this 
effort  will  be  limited  to  the  use  of  management  reserve  funds 
to  cover  the  extra  costs  due  to  uncertainty. 

Limitations 

The  accuracy  of  this  research  was  limited  by  the 
extent  to  which  people  were  willing  to  share  information 
concerning  this  very  sensitive  issue.  Past  literature  has 
been  incomplete  in  its  coverage  of  management  reserves, 
although  models  to  assess  risk  and  uncertainty  were  covered 
in  greater  detail. 

Research  Questions 

Three  research  questions  in  addition  to  those  raised 
in  the  problem  statement  were  posed? 

1.  Do  management  reserves  need  to  he  visible? 

2.  Should  more  approaches  to  estimate  risk  and 
uncertainty  be  developed  and/or  should  a  bett-r  management 
system  be  implemented? 

3-  Is  this  problem  hiding  another  issue  which  has 
not  been  openly  discussed? 
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CHAPTER  2 


METHODOLOGY 


Overview 

Since  the  objective  of  the  research  effort  was  to 
better  identify  the  issues  concerning  the  use  of  management 
reserves  to  budget  for  the  uncertainty  inherent  in  any 
project,  the  Research  Question  Research  Method  approach  was 
selected.  This  type  of  research  is  concept-building  in 
order  to  expand  the  "frontiers  of  knowledge"  (33)  when  any 
of  the  following  exist: 

a.  A  hypothesis  cannot  be  qualitatively  or  quanti¬ 
tatively  tested; 

b.  Sufficient  information  has  not  been  generated 
to  render  a  reasonable  hypothesis; 

c.  Synthesis  is  appropriate  and  justified  (39). 

The  first  step  was  to  conduct  a  preliminary  litera¬ 
ture  review  to  evaluate  whether  DOD  was  experiencing  prob¬ 
lems  in  dealing  with  risk  and  uncertainty.  upon  validating 
that  there  was  a  problem,  another  review  similar  to  the 
first  was  conducted  to  identify  sources  of  information  con¬ 
cerning  management  of  risk  by  the  use  of  management  reserve 
techniques.  Little  information  was  found  and  the  results  of 
the  two  literature  reviews  were  used  to  formulate  the  proble 
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statement ,  define  terms,  and  define  research  objectives  as 
well-  as  give  background  information  concerning  the  problem 
(39). 

Once  the  literature  objectives  were  defined,  a  more 
in-depth  literature  review  was  conducted  in  order  to  review 
any  publications  concerning  DOD  management  of  risk  and  uncer¬ 
tainty  in  R&D.  This  resulted  in  data  which  could  then  be 
analyzed  in  order  to  fulfill  the  research  objectives  and 
further  identify  the  factors  which  bear  on  the  problem.  The 
data  was  analyzed  to  see  if  any  trends  in  the  management  of 
risk  and  uncertainty  could  be  identified.  The  various 
methods  of  handling  risk  and  uncertainty  were  also  reviewed 
to  see  if  any  were  adequate  to  predict  the  amount  of  money 
needed  to  cover  the  risk  and  uncertainty  in  a  project.  If 
none  were  found,  the  next  objective  was  to  suggest  ways  to 
appropriately  handle  risk  and  uncertainty  or  suggest  further 
areas  of  research  based  on  the  analysis  (39). 

In  order  to  verify  that  the  problem  was  current  and 
to  verify  the  findings  of  the  literature  review.  Interviews 
with  practicing  professionals  in  the  fields  of  cost  estimat¬ 
ing,  cost  analysis,  and  project  management  at  various  levels 
of  command  were  conducted.  This  was  also  used  in  the  anal¬ 
ysis  of  the  data  collected  in  the  in-depth  literature  review. 
The  two  leading  methods  used  by  the  U.  S.  Air  Force  (USAF) 
and  the  U.  S.  Army,  Identified  during  the  above  process, 
were  analyzed  and  the  examples  given  In  the  documents 
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describing  each  method  are  presented  in  Chapter  4  and  the 
Appendices.  This  method  of  illustration  was  selected  since 
any  data  from  an  actual  completed  project  would  not  show  a 
complete  breakdown  of  the  risk  and  uncertainty.  The  inter¬ 
view  results  are  presented  in  Chapter  3  following  the  dis¬ 
cussion  of  the  literature  review.  Chapter  5  contains  a  sum¬ 
mary  discussion  of  the  research  effort  as  well  as  conclusions 
and  recommendations. 

Literature  Review 

A  search  of  recent  periodicals  was  made.  The  main 
thrust  of  the  search  was  conducted  by  submitting  various 
key  words  to  the  Defense  Technical  Information  Center  (DTIC). 
The  following  terms  were  used  to  obtain  Technical  Report 
Summaries  (bibliographies). 

Technical  Report  Summary  Number  1.  Air  Force  Bud¬ 
gets,  Army  Budgets,  Budgets,  Federal  Budgets,  Military  Bud¬ 
gets,  and  Naval  Budgets  were  each  compared  with  the  more 
restrictive  term  Air  Force  Research  with  the  emphasis  on 
Cost,  Cost  Analysis,  Cost  Effectiveness,  Cost  Models,  Cost 
Overruns,  Costs,  Design  to  Cost,  High  Costs,  Indirect  Costs, 
Life  Cycle  Costs,  Low  Costs,  and  Price  Index. 

Technical  Report  Summary  Number  2.  Cost  Analysis, 
Cost  Estimates,  and  Cost  Overruns  were  cross  referenced 
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with  the  terms  Management  Planning  and  Control,  Military  Bud¬ 
gets,  and  Planning  Programming  Budgeting. 

Technical  Report  Summary  Number  3»  Cost  Models  were 
cross  referenced  with  the  terms  Management  Planning  and  Con¬ 
trol,  Military  Budgets,  and  Planning  Programming  Budgeting. 

Technical  Report  Summary  Number  *1.  Risk  and  Risk 
Analysis  were  cross  referenced  with  the  term  Forecasting. 

The  results  of  searching  through  these  bibliogra¬ 
phies  were  forty-five  possible  candidates  to  be  further 
studied  to  see  if  any  information  relative  to  the  research 
objectives  could  be  found.  The  results  of  this  effort  are 
found  in  Chapter  3  of  this  report. 

Interviews 

In  order  to  get  the  latest  information  concerning  the 
treatment  of  risk  and  uncertainty,  personal  Interviews  were 
conducted.  A  semi-structured  approach  was  used.  Before  the 
interviews,  a  list  of  questions  and  objectives  was  prepared 
based  on  the  data  obtained  in  the  literature  reviews.  The 
questions  were  prepared  with  the  purpose  of  verifying  that 
the  data  found  in  the  literature  review  was  still  current 
and  that  there  was  nothing  new  that  had  been  missed.  The 
main  objective  was  not  only  to  verify  the  data  but  also  to 
allow  the  Individuals  contacted  to  freely  express  their  opin¬ 
ions  without  being  unduly  influenced  by  the  objectives  of 
the  research. 
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The  individuals  were  then  to  be  contacted,  and  after 
a  standardized  brief  introduction  concerning  the  purpose  of 
the  interview,  were  to  be  allowed  to  comment  as  they  saw  fit 
until  they  had  discussed  all  that  they  felt  of  worth.  At 
that  time,  if  the  objectives  and  questions  had  not  been  cov¬ 
ered,  these  would  be  brought  up.  It  was  felt  that  this 
approach  would  allow  a  free  flow  of  information,  unhampered 
by  the  resistance  that  structure  can  sometimes  produce. 

Once  the  individual  completed  their  presentation,  the  struc¬ 
tured  questions  could  be  used  to  clarify  any  issues  not  cov¬ 
ered  or  understood. 

In  conducting  an  interview  in  such  a  controversial 
area,  it  was  felt  that  some  individuals  could  feel  like  one 
was  prying  into  areas  that  might  cause  political  repercus¬ 
sions  if  revealed.  Therefore,  the  introduction  was  impor¬ 
tant  in  establishing  the  purpose  and  ground  rules  of  the 
interview.  The  interviewee  was  assured  that  any  sensitive 
information  would  not  be  disclosed  nor  would  be  printed  in 
this  report.  It  must  be  remembered  that  In  an  area  as  con¬ 
troversial  and  politically  sensitive  as  the  use  of  manage¬ 
ment  reserves  and  evaluating  risk,  one  cannot  give  official 
statements  on  matters  still  in  the  process  of  resolution  by 
the  Congress,  DOD,  and  other  governmental  departments  and 
officials.  The  results  of  the  Interviews  will  be  found  in 
Chapter  3. 
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Analysis  of  Data  Gathered 


Analysis  of  the  references  found  in  the  literature 
reviews  leads  to  the  identification  of  two  leading  techniques 
for  possible  DOD  risk  analysis,  one  used  by  the  U.  S.  Army 
and  the  other  by  the  USAF.  The  documents,  containing  a  des¬ 
cription  of  each  of  the  techniques,  contained  an  example 
which  included  both  the  input  data  and  the  output  data. 

Both  techniques  are  evaluated,  with  advantages  and  disad¬ 
vantages  for  each.  Examples  of  both  are  presented  in  the 
Appendices.  The  numbers  presented  are  for  clarification 
only  and  do  not  represent  a  real  project. 

Conclusions /Recommendations 

Chapter  5  will  present  a  summary  of  t'ne  findings 
and  any  conclusions  reached  as  a  result  of  the  research 
efforts.  Recommendations  will  also  be  presented  as  to  the 
possible  use  of  the  research  and  suggestions  for  further 
study . 
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CHAPTER  3 


LITERATURE  REVIEW/ INTERVIEWS 


Overview 

Formal  recognition  and  emphasis  on  risk  and  uncer¬ 
tainty  analysis  in  DOD  [Department  of  Defense]  resulted 
from  a  31  July  69  memorandum  from  David  Packard,  Deputy 
Secretary  of  Defense;  to  the  Military  Departments  iden¬ 
tifying  problem  areas  in  the  weapon  system  acquisition 
process.  In  this  memo,  Secretary  Packard  cited  inade¬ 
quate  identification  and  consideration  of  risks  in 
major  programs  as  a  problem  area  requiring  action  and 
that  risk  analysis  be  applied  by  program  managers  in 
their  daily  activities.  .  .  .  The  past  decade  has 
also  witnessed  an  explosion  of  books  and  technical 
papers  on  risk  and  uncertainty  analysis  [34:12]. 

A  1969  Rand  report  suggested  that  their  analysis  of 
cost  estimates  for  the  1960s  showed  a  25%  less  optimistic 
bias  than  those  programs  of  the  1950s  (28:vi).  This  bias 
refers  to  cost  estimates  that  do  not  reflect  the  actual 
amount  that  Is  required  for  completion  of  a  project.  These 
cost  estimates  are  called  optimistic  since  they  reflect  a 
cost  that  one  would  like  to  occur  rather  than  what  will 
actually  occur.  This  suggested  that  as  each  decade  has 
passed,  cost  estimates  have  become  more  accurate  and  more 
"realistic".  The  Rand  report  concluded  that 

The  ability  of  the  Government  to  make  programmed 
and  actual  program  outcomes  [cost,  performance,  etc.  ] 
coincide  has  been  of  considerable  concern  to  Congress, 
the  military  departments,  and  the  public  [23:^1]. 

Although  there  was  a  demand  for  better  cost  estimating 


techniques,  the  Rand  Corporation  found  it  necessary 

to  point  out  that  the  data  suggest  that  estimating 
reforms  are  riot  likely  to  resolve  the  underlying  prob¬ 
lems.  Not  enough  is  known  about  the  causes  of  cost 
escalation  to  support  the  contention  that  estimating 
errors  are  the  major  contributors  [28:42], 

or,  looking  at  the  other  side  of  the  coin,  that  Improved 
tracking  of  actual  costs  with  cost  estimates  could  be  attrib¬ 
uted  solely  to  improved  cost  estimating. 

Leggitt,  of  Booz,  Allen  and  Hamilton,  stated  that 
prior  to  19^1, 

the  Defense  Department  developed  its  force  struc¬ 
ture  by  starting  with  a  budget  and  then  seeking  out 
possible  programs.  There  was  no  pragmatic  way  of 
relating  costs  to  weapons  systems,  tasks  and  missions 
[7:1-2]. 

The  period  following  the  end  of  the  1960s  saw  the  increased 
use  of  risk  and  uncertainty  analysis  with  cost  estimating 
techniques.  Rich,  of  the  Rand  Corporation,  found  after  com¬ 
paring  programs  from  the  1960s  with  programs  of  the  1970s, 
that  the  1970s  were  "coming  closer  to  their  cost  goals  by 
some  10  to  20  percentage  points  [34:172]."  He  also  made  the 
observation  that  there  was  strong  evidence  that  a  substan¬ 
tial  part  of  the  cost  growth  in  the  government  acquisition 
process  was  beyond  the  control  and  responsibility  of  the  pro¬ 
gram  managers,  and  at  times  outside  the  control  of  top  level 
acquisition  managers  in  OSD  (Office  of  the  Secretary  of 
Defense)  as  well  as  the  services  (34:172).  In  fact,  he  said 
"that  the  search  for  better  cost  control  should  include 
consideration  of  changes  in  government  policies  and 
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procedures  outside  the  Department  of  Defense  [39:172]." 

In  1981,  Colonel  Martin,  of  the  Air  Force  Busir ess 
Research  Management  Center,  pointed  out  that 

An  assessment  of  uncertainty  is  critical;  however, 
rigorous  and  formal  treatment  of  this  element  has  not 
been  an  integral  part  of  management's  Information  sys¬ 
tem.  Only  recently  have  serious  efforts  been  attempted 
to  structure  the  process  of  analyzing  the  uncertainty 
which  confronts  managers  responsible  for  the  develop¬ 
ment  contract  and  the  related  program  [39:178]. 

Today's  manager  has  to  contend  with  many  factors  which  can 
cause  the  total  program  costs  to  exceed  his  initial  estimate. 
These  factors  will  be  presented  in  the  next  section,  cate¬ 
gories  of  risk  and  uncertainty.  It  can  be  said  at  this  time 
though,  that  these  factors  have  caused  large  cost  overruns. 
For  example,  Sherman  and  Gardiner  found  that 

unexpected  high  inflation,  supply  and  demand  factors, 
poor  resource  allocation  and  managerial  inefficiency  can 
cause  total  estimated  program  costs  at  completion  to 
exceed  early  estimates  by  10,  100,  or  even  300  percent 
[39:209]. 

There  are  many  tools  and  techniques  available  to  aid  the  pro¬ 
gram  manager,  but  "Obviously,  no  set  of  tools  or  techniques 
is  a  substitute  for  [good]  management  .  .  .  [39:58]." 

In  considering  the  factors  to  be  presented  in  the 
following  section,  it  should  be  remembered  that  some  factors 
are  project  dependent  and  that  there  is  not  enough  space  in 
this  report  co  include  all  of  them.  Only  those  which 
appeared  to  apply  to  all  projects  and  were  mentioned  in 
numbers  of  articles  were  included  in  this  report. 

The  next  five  sections  cover:  Categories  of  Risk  and 
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Uncertainty;  Uncertainty  and  Risk  Management;  Cost  Estimat¬ 
ing  Techniques;  Uncertainty  and  Risk  Techniques;  ar.d  Inter¬ 
view  Results.  The  Cost  Estimating  Techniques  have  been 
placed  before  Uncertainty  and  Risk  Techniques  because  a 
brief  discussion  of  cost  estimation  will  aid  the  reader  in 
understanding  the  need  for  uncertainty  and  risk  techniques. 
The  section  on  Uncertainty  and  Risk  Management  will  provide 
further  information.  The  Interview  Results  are  presented 
in  the  last  section.  No  names  have  been  attached  to  the 
results,  for  reasons  given  previously.  In  conducting  these 
interviews,  the  researcher  found  that  those  people  inter¬ 
viewed  were  willing  to  discuss  the  matter,  but  did  not  want 
to  be  identified,  since  government  policy  on  this  matter  is 
still  being  determined.  It  was  found  that  within  the  DCD 
structure  the  U.  S.  Army's  Total  Risk  Assessing  Cost  Estimate 
(TRACE)  and  the  U.  S.  Air  Force  Cost  Uncertainty/Risk  Anal¬ 
ysis  Model  (described  in  detail  in  Chapter  4)  were  the  lead¬ 
ing  methods  proposed  as  fulfilling  the  requirements  of  the 
Carlucci  initiatives. 

Categories  of  Risk  and  Uncertainty 

Before  describing  risk  and  uncertainty  management, 
cost  estimating  techniques  and  risk  and  uncertainty  tech¬ 
niques,  a  brief  discussion  of  the  categories  of  risk  and 
uncertainty  follows.  This  discussion  will  also  include  a 
brief  description  of  some  of  the  risk  factors  that  are 
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related  to  these  categories.  It  should  be  emphasized  at 
this  point  that  not  all  of  the  risk  factors  can  be  presented. 
There  are  many  different  factors  which  can  be  identified  in 
any  system.  This  literature  review  presents  those  factors 
which  seemed  most  prevalent.  They  have  been  presented  here 
since  these  are  some  of  the  factors  which  a  manager  must  con¬ 
sider  when  evaluating  various  management  principles  and  tech¬ 
niques  to  be  used  in  the  management  of  a  particular  program. 
The  factors  which  have  been  found  to  be  a  major  cause  of 
uncertainty  should  be  given  the  most  emphasis,  but  not  all 
of  the  emphasis,  when  planning  a  new  program.  Carter  studied 
and  analyzed  historical  cost  data  and  indicated  that 

technological  uncertainty  is  a  major  cause  of  pro¬ 
duction  and  development  cost  overruns.  .  .  .  Require¬ 
ments  and  technological  uncertainties  are  the  primary 
causes  of  errors  In  the  cost  estimates  of  future  weapon 
systems  [4:27]. 

He  made  this  statement  in  1965  and  described  eleven  factors  . 
which  contribute  to  cost  estimating  errors.  Tables  1  and 
2  list  the  factors,  authors,  and  the  categories  under  which 
the  factors  have  been  placed. 

Several  categories  of  risk  and  uncertainty  have  been 
Identified.  Beverly  et  al.  listed  three  types  of  uncertainty 
"Design  and  Technology  uncertainty;  Uncertainty  regarding  the 
ability  of  the  contractor  to  meet  the  terms  of  the  contract 
(scheduling);  and  Cost  uncertainty  [34:265]." 

Three  categories  of  risk  identified  by  Worm  are: 
technical  risk,  cost  risk,  and  schedule  risk  (34:97)  which 
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TABLE  1 
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are  quite  similar  to  those  described  by  Beverly  et  al. 

These  categories  apparently  were  derived  from  cost,  schedule 
and  performance  which  are  described  in  life  cycle  costing 
and  Cost /Schedule  Control  System  Criteria  (C/SC SC).  Cost, 
schedule,  and  performance  were  identified  at  the  beginning 
of  the  1970s. 

Worm’s  categories  give  the  best  description  of  risk. 
He  stated  that  ’’The  technical  and  schedule  risks  are  impor¬ 
tant  factors  in  the  estimation  of  cost  risk  .  .  .  [3*1:97]." 
Technical  risks  (related  to  performance  risks) ( 27:97-115) 
are  derived  from  a  knowledge  of  the  probability  density 
function  that  describes  the  technical  uncertainty  in  an  R&D 
program  which  includes:  technical  feasibility,  selection  of 
the  best  technical  approach,  correct  formulation  of  the  prob 
lem,  correct  choice  of  the  design  parameters  including  a 
proper  parametric  analysis  (includes  performance  factors 
such  as  kill  radius,  speed,  range,  payload,  etc.),  and  cor¬ 
rect  trade-off  analysis  (i.e.,  the  trade-off  "between  design 
speed  and  performance,  between  design  payload  and  perfor¬ 
mance,  and  so  on  [27:113]").  Schedule  risks  are  derived 
from  a  knowledge  of  the  probability  density  function  that 
describes  schedule  uncertainty  (timing)  in  an  R&D  program 
which  includes:  delays  in  production,  late  delivery  of 
parts  or  materials,  and  mistiming  of  the  arrival/completion 
of  sequential  assembled  parts  (32:61).  These  relate  to  cost 
risk  in  that  a  probability  density  function  can  be  derived 
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which  represents  the  combination  of  the  technical  and  sched¬ 
ule  risks.  The  cost  uncertainty  includes  the  cost  variances 
that  would  result  from  the  effect  of  technical  and  schedule 
uncertainty  on  labor  and  overhead  costs,  material  costs,  and 
unforeseen  penalty  costs  (32:61). 

It  becomes  necessary  at  this  point  to  further  de¬ 
scribe  some  of  the  factors  which  cause  cost  growth  since  they 
are  also  those  factors  which  contain  an  element  of  uncer¬ 
tainty.  These  factors  can  be  used  to  obtain  a  risk  estimate 
if  enough  historical  data  is  gathered  so  that  a  probability 
density  function  can  be  derived.  It  is  these  factors  which 
must  be  managed  as  well  as  other  factors  which  arise  such  as 
top  level  directed  program  changes,  budget  cuts,  and  other 
unforeseen  changes  in  the  political  climate.  These  factors 
must  not  only  be  included  in  estimating  a  cost  of  a  program 
but  are  also  necessary  in  the  control  of  cost  growth  of  a 
program  and/or  risk  management. 

In  1965,  R.  D.  Carter  (4)  presented  an  analysis  of 
the  factors  which  influence  cost  estimate  reliability  and 
suggested  that  cost  sensitivity  analysis  was  the  best  method 
to  examine  the  changes  in  the  total  system  cost  when  these 
factors  were  varied  over  their  relevant  range  (4:43).  He 
stated  that 

The  incongruity  between  estimated  and  actual  costs 
of  today's  weapon  systems  Indicates  a  need  for  cost 
estimates  which  more  accurately  predict  the  cost  of 
future  weapon  systems.  Estimates  made  near  the 
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beginning  of  a  development  program  are  particularly 
unreliable  [4:iii]. 

He  went  on  to  explain  that,  without  all  the  necessary  ele¬ 
ments  of  the  total  system  included,  the  cost  estimate  would 
be  inadequate  (4:iii).  The  factors  presented  which  cause 
cost  estimate  unreliability  are  as  follows: 

1.  Technical  Complexity 

2.  Technical  Obsolescence 

3.  Schedule  Slippage 

4.  Timing  of  Cost  Estimate 

5.  Competitive  Optimism 

6.  (This  paragraph  deleted  from  documents  for  open 
publication.) 

7.  Inaccurate  Cost  Estimates 

8.  Price  and  Procurement  Levels 

9.  Cost-Estimating  Uncertainty 

10.  Air  Force  Requirements  Uncertainty 

11.  Technological  Uncertainty  [4:10-27] 

The  above  factors  are  not  easy  to  categorize  under 
cost  risk,  schedule  risk,  and  performance  risk  since  some  of 
them  affect  more  than  one  category.  This  is  not  surprising 
since  the  categories  are  not  Independent  of  each  other.  For 
example,  a  change  in  performance  requirements  can  lead  to  a 
change  in  schedule,  which  will  cause  a  change  in  costs.  If 
the  performance  risk  has  been  underestimated,  it  could 
result  in  an  underestimation  of  the  schedule  risk  if  the 
analyst  used  the  performance  characteristics  to  determine  the 
schedule.  Both  affect  the  cost  risk.  Carter's  eleven  fac¬ 
tors  are  described  below. 

Technical  Complexity  includes  a  large  number  of 
technical  problems,  the  interrelation  of  the  technical  pro- 
lems,  and  the  large  number  of  components  when  considered 
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from  the  point  of  reliability  (4:10).  An  example  of  techni¬ 
cal  interrelation  could  be  the  discovery  that  a  newly 
designed  cockpit  display  instrument  will  not  fit  into  the 
instrument  panel.  In  order  to  fit  it  into  the  panel,  the 
other  instruments  must  be  moved  which  will  necessitate  the 
redesign  of  the  instrument  panel  and  possibly  the  rerouting 
of  electrical,  pneumatic,  and  hydraulic  lines.  As  any  sys¬ 
tem  grows,  reliability  can  change  since  there  are  more 
parts  that  can  fail. 

Technical  Obsolescence  occurs  when  the  state-of-the- 
art  advances  beyond  the  present  day  level  (4:10).  An  example 
could  be  the  selection  of  vacuum  tube  operated  components 
in  the  design  of  a  new  avionics  system  at  the  time  transis¬ 
torized  components  were  just  appearing  on  the  market. 

Schedule  Slippages  involve  a  change  in  the  program 
milestones.  These  changes  could  result  In  stretching  out 
the  time  required  to  complete  the  program.  It  could  possibly 
Include  compressing  (shortening)  the  schedule  time  If  the 
customer  wanted  the  project  completed  earlier  than  originally 
planned. 

The  Timing  of  the  Cost  Estimate  can  affect  overruns, 
since  more  elements  are  better  defined  for  later  estimates. 

Competitive  Optimism  results  in  estimates  of  sched¬ 
ule  and  costs  which  are  less  than  what  it  would  actually 
take  to  obtain  a  given  set  of  performance  characteristics. 
Competitive  Optimism  involves  both  the  contractor  and  the  - 
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U.  S.  Air  Force,  since  the  contractor  wants  to  obtain  the 
contract  while  the  Air  Force  wants  to  obtain  a  new  weapon 
system.  "The  contractor  usually  reflects  his  desire  to  win 
the  contract  by  outbidding  the  cdmpetition  with  optimistic 
cost  estimates  for  initial  development  [4:14].”  The  Air 
Force  may  either  discourage  realistic  cost  estimates  because 
of  an  inadequate  budget  or  because  they  place  a  premium  on 
maximum  advancement  of  the  state-of-the-art:  the  result, 
optimistic  technical  proposals  and  optimistic  cost  estimates 
(4:14). 

Inaccurate  Cost  Estimates  may  be  a  result  of  an 
incomplete  cost  analysis.  "Many  times,  the  estimate  of  total 
system  costs  does  not  include  all  the  necessary  elements  of 
system  costs  [4:17]."  Price  and  Procurement  Levels,  while 
lumped  together  In  Carter’s  article,  were  of  unequal  strength 
as  sources  of  error. 

Changes  In  price  levels  during  the  development  of 
a  weapon  system  are  relatively  small  sources  of  error, 
and  the  cost  estimates  are  usually  presented  in  [con¬ 
stant]  dollar  values  [dollars  adjusted  for  the  effects 
of  Inflat Ion ][ 4: 17]. 

Changes  in  procurement  level  (quantities  bought)  caused  the 
largest  cost  overruns  of  the  two. 

Cost-Estimating  Uncertainty  is  a  result  of  the  inac¬ 
curacies  caused  by'  the  following:  the  cost  functions  not 
fitting  the  data  exactly,  extrapolations  beyond  the  range 
of  the  sample  data,  and  aggregation  of  data  at  a  higher 
lfevel. 
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Cost-estimating  relations  used  in  cost  analysis 
cannot  be  assumed  to  hold  exactly.  In  estimating  a 
certain  cost  component  as  a  function  of  many  variables, 
it  cannot  be  assumed  that  the  variables  will  predict 
the  particular  cost  with  certainty.  .  .  .  Cost¬ 
estimating  relations  from  past  experience  are  some¬ 
times  used.  Extrapolations  beyond  the  range  of  the 
sample  or  data  base  from  which  the  estimating  rela¬ 
tion  was  derived  are  another  source  of  uncertainty. 
Cost-estimating  errors  may  arise  because  of  the  use 
•  of  techniques  which  involve  a  considerable  amount  of 
aggregation.  An  important  cost  element  may  be  over¬ 
looked.  .  ■.  .  Large  aggregations  also  make  cost 
comparisons  of  alternatives  difficult  [4:19]. 

Air  Force  Requirements  Uncertainty  addresses  "sud¬ 
den  and  unpredictable  changes  in  technology,  enemy  plans, 
and  U.  S.  defense  policies  which  might  require  program  redi¬ 
rection  or  even  cancellation  [4:19-20]."  If  the  program  is 
redirected,  two  kinds  of  configuration  changes  could  evolve: 
hardware  characteristics,  or  the  system’s  operational  con¬ 
cept  (4:20).  Either  will  cause  an  unknown  increase  in  the 
cost  over  the  original  estimate. 

Technological  Uncertainty  covers  the  many  problems 
which  can  occur  as  technology  develops  or  changes.  Whether 
they  will  occur  is  uncertain.  "...  each  of  these  unex¬ 
pected  difficulties  has  an  effect  on  the  contractor’s  ability 
to  meet  original  time,  quality,  and  cost  predictions  [4:21]." 

In  1981,  Defense  Advanced  Research  Projects  Agency 
(DARPA)  researchers  identified  the  following  seven  factors 
as  contributors  to  cost  increases: 

1.  Quantity — changes  including  scope. 

2.  Engineering — changes  altering  a  system’s  estab¬ 
lished  physical  or  functional  characteristics. 
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3.  Support — changes  involving  spare  parts,  ancil¬ 
lary  equipment,  warranty  provisions,  and  government- 
furnished  property  or  equipment. 

4.  Schedule — changes  in  delivery  schedules,  com¬ 
pletion  date  or  some  intermediate  milestone  of  develop¬ 
ment,  production  or  construction . 

5.  Economic — changes  that  are  influenced  by  one  or 
more  factors  in  the  economy,  such  as  inflation. 

6.  Estimating--corrections  or  other  changes  occur¬ 
ring  since  the  initial  or  .  .  .  baseline  estimates  for 
program  or  project  costs. 

7.  Sundry — changes  other  than  tne  above  categories, 
such  as  environmental  cost  and  relocation  assistance 
for  water  and  highway  projects  [9:1-5]. 

The  above  seven  factors  are  identical  to  the  Cost  Variance 

Analysis  Breakout  found  in  the  SAR  (Selected  Acquisition 

Report)  except  that  the  term  "Other"  is  found  in  place  of 

"Sundry"  in  the  SAR  (1;  13). 

Not  only  have  there  been  attempts  to  categorize  the 
factors  which  affect  risk,  but  there  have  also  been  attempts 
to  categorize  uncertainty.  Rowe  and  Somers  quoted  Martin 
D.  Martin's  PhD  dissertation  which  listed  four  basic  cate¬ 
gories  of  uncertainty:  Environmental,  Functional,  Informa¬ 
tional,  and  Technical.  Environmental  uncertainty  includes 
such  factors  as  social  and  political  effects,  factors 
external  to  the  project,  communication  problems,  and  time. 
Functional  uncertainty  includes  technological  uncertainty, 
production  inadequacies ,  and  financial  and  business  problems. 
Informational  uncertainty  can  be  attributed  to  the  unknowns 
found  in  Incomplete  knowledge,  whether  anticipated  or  unfore¬ 
seen  (  3-4:33) . 

vc 
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A.  J.  Rowe  and  I.  A.  Somers  also  Identified  four 


uncertainty  categories:  Internal  Program  Uncertainty,  Tech¬ 
nical  Uncertainty,  Process  Uncertainty,  and  Target  Uncer¬ 
tainty.  They  described  their  four  basic  variables  or  cate¬ 
gories  as  equivalent  to  those  developed  in  a  previous  USAF 
Academy  study.  The  Rowe  and  Somers'  term  is  inserted  in 
brackets  next  to  the  USAF  Academy  term  of  the  same  meaning 
and  definition. 

1.  Internal  Program  Uncertainty  [Internal  Control]: 
Deals  with  the  way  in  which  the  program  is  organized, 
planned  and  managed.  Uncertainty  of  the  initial  esti¬ 
mate  and  its  Impact  on  program  management.  Uncertainty 
in  the  acquisition  strategy  and  outcome.  Uncertainty 

in  resources  needed,  flexibility,  or  lack  of  contin¬ 
gency  plans.  Competing  demands,  including  conflict 
between  reliability,  vulnerability  and  maintainability 
with  performance  and  operating  costs. 

2.  Technical  Uncertainty  [Technological  Uncertainty 
Covers  the  feasibility  of  developing  the  system  at  all, 
including  the  degree  of  technical  difficulty.  Generally 
starts  with  an  optimistic  estimate  of  the  state-of-the- 
art  and  often  leads  to  a  slippery  technical  baseline. 

3.  Process  Uncertainty  [Customer  Uncertainty]: 

Deals  with  the  sensitivity  to  changes  in  the  external 
environment  such  as  changes  In  priorities  or  policies, 
the  president's  budget,  congressional  political  con¬ 
siderations,  etc.  Unavailability  of  funding/resources 
when  needed.  Uncertainty  in  criteria  used  for  changes, 
control,  surveillance,  DSARC  decisions,  etc.  Effects 
of  inflation  and  government  regulation. 

4.  Target  Uncertainty  [Environmental  Uncertainty]: 
Covers  the  uncertainty  in  meeting  performance,  cost  or 
schedule  goals  and  determination  of  needs.  Uncertainty 
in  translating  abstract  needs  into  concrete  specifica¬ 
tions.  Problems  of  early  estimates  which  are  seldom 
revised  [3^:26]. 

The  next  section  will  discuss  the  management  o.  risk  and 
uncertainty . 


30 


Uncertainty  and  Risk  Management 


Management  policy  must  be  developed  which  recognizes 
the  fact  that  there  are  many  tools  involved  in  controlling  a 
program.  One  tool  for  controlling  a  program  is  forecasting 
which  is  the  most  important  function  of  program  control. 

"It  is  looking  downstream  to  see  where  a  program  is  going 
and  proposing  alternatives  to  accommodate  threats  and  hasten 
progress  (AFSCP  800-3,  p.  6 . 2 )  [  5  :  22] . There  are  many  areas 
which  a  program  manager  must  consider  such  as  selection  of 
personnel,  project  organization,  interfaces  with  other  agen¬ 
cies,  etc.  He  must  also  consider  the  risk  and  uncertainty 
involved  in  his  project.  Therefore,  this  section  will  dis¬ 
cuss  some  of  the  techniques  of  management  of  forecasting  or 
cost  estimating  with  the  attendant  analysis  of  risk  and 
uncertainty.  Cost  estimating  techniques  and  uncertainty  and 
risk  techniques  will  be  covered  in  the  following  two  sections. 

The  factors  described  in  the  last  section  can  be 
used  to  obtain  a  management  reserve  estimate  if  enough  his¬ 
torical  data  can  be  derived.  If  a  density  function  must  be 
imputed,  we  are  then  dealing  with  uncertainty.  If  we  can 
take  the  time  and  make  the  effort  necessary  we  may  be  able 
to  derive  a  probability  distribution  and  thus  be  dealing 
with  risk.  It  is  these  factors  which  must  be  managed  as  well 
as  other  noncont reliable  events  which  arise,  such  as  top 
level  directed  program  changes,  budget  cuts,  and  other 
unforeseen  changes  in  the  political  climate.  These  factors 


must  not  only  be  included  in  estimating  a  cost  of  a  program 
but  are  also  necessary  in  the  control  of  cost  growth  of  a 
program,  which  of  course  includes  cost  growth  due  to  uncer¬ 
tainty  and  risk. 

The  government  has  initiated  many  programs  and  poli¬ 
cies  to  improve  their  control  of  risk  and  uncertainty  while 
at  the  same  time  reducing  their  effects.  In  looking  through 
the  indexes  of  the  various  Department  of  Defense  Directives 
and  Instructions,  the  term  "management  reserves"  could  not 
be  found.  Money  used  to  cover  risk  and  uncertainty  was  found 
in  the  engineering  change  order  (ECO)  line  of  a  budget. 

The  term  "unencumbered  funds"  was  another  term  sometimes 
used  to  delineate  money  set  aside  to  cover  those  changes 
which  are  a  result  of  unforeseen  circumstances.  Cost/Sched- 
ule  Control  Systems  Criteria  (C/SCSC,  contained  in  DOD 
Instruction  700G.2,  Performance  Measurement  for  Selected 
Acquisition,  and  DOD  7000.10,  Contract  Cost  Performance, 

Funds  Status  and  Cost /Schedule  System  Reports)  was  the  only 
policy  found  that  allowed  management  reserve.  The  contractor 
is  allowed  to  set  aside  a  portion  of  the  funds  he  receives 
as  a  management  reserve  and  give  his  shops  lower  budgets  in 
order  to  exercise  proper  management  control  of  his  work 
force.  If  an  ECO  change  is  made,  the  contractor  receives 
more  money  which  he  can  add  to  his  management  reserve. 

i 

In  1973,  El-Sabban  described  C/SCSC  and  suggested 
that  the  Information  obtained  from  the  contractor's  monthly 
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progress  report  could  be  used  to  forecast  future  cost  and 
estimate  changes.  He  stated  that  "The  adoption  of  the  laird- 
Packard  principle  of  closer  ties  with  the  contractor  and  more 
efficient  supervision  of  the  project  operation  throughout  the 
life  of  the  contract  [14:11]"  lead  to  the  establishment  of 
C/SCSC.  The  monthly  cost  performance  report  contained  a  work 
breakdown  structure  summary  which  allowed  the  program  manager 
to  assess  the  contractor’s  adherence  to  the  planned  cost  and 
schedule.  The  program  manager  could  figure  the  variance  for 
each  item  in  the  work  breakdown  structure  and  monitor  the 
progress  of  the  project  compared  to  what  was  originally 
planned. 

Although  C/3CSC  was  set  forth  as  a  government  policy, 
it  did  not  function  perfectly.  Feller  reported  in  1978  on 
a  study  conducted  by  the  Office  of  Naval  Research  to  improve 
the  effectiveness  of  the  Navy’s  systems  acquisition  process. 
The  conclusions  showed  that  the  Navy  would  have  to  add  addi¬ 
tional  information  to  DODI  7000.2  in  order  to  overcome  prob¬ 
lems  found  in  applying  the  instruction  to  Navy  shipbuilding 
project  management  (3:506-510).  They  found  that 

The  contractor's  cost/schedule  control  system  does 
not  provide  adequate  data  upon  which  to  base  a  mean¬ 
ingful  "forward  look"  .  .  .to: 

(a)  Identify  potential  problems 

(b)  Evaluate  contractor’s  work  load  and  work  force 
planning 

(c)  Evaluate  cost/schedule  Impacts  of  change  orders 
or  other  contract  changes. 


The  contractor's  use  of  a  "deterministic"  (as  opposed 
to  "probabilistic" )  scheduling  approach  results  in 
schedules  which  are  inherently  optimistic  [3:509]. 

It  was  also  found  that  not  only  did  the  contractor  fail  to 
furnish  the  Navy  with  information  concerning  work  order 
sequence  so  that  logical  predecessor-successor  relationships 
could  be  established,  but,  also,  actual  start  and  finish 
dates  for  all  work  orders  and  rescheduled  activities  were 
not  always  reported  (3:509-510).  All  this  suggests  that  not 
only  must  each  service  tailor  any  DOD  policies  and  direc¬ 
tives  to  its  own  needs,  but  that  it  is  each  service's 
responsibility  to  insure  that  the  proper  management  policies 
are  in  place  to  enable  the  program  manager  to  obtain  the 
necessary  data  required  for  an  accurate  cost  estimate  which 
can  take  into  account  risk  and  uncertainty. 

In  November  1969,  the  Rand  Corporation  published  the 
results  of  their  research  which  evaluated  the  accuracy  of 
the  predictions  of  "recent"  military  program  cost  and  article 
performance  outcomes  as  well  as  a  comparison  of  the  ratios 
of  actual-to-predicted  outcomes  for  1950s  to  1960s  programs. 
They  found  that  the  required  information  needed  for  their 
research  was  not  easily  obtained.  In  fact. 

Various  data  retrieval  systems  exist,  but  the  under¬ 
lying  problems  of  defining,  categorizing,  and  under¬ 
standing  program  growth  are  accentuated  by  .  .  .  ambi¬ 
guities,  and  uncertainties  In  the  data.  Many  of  the 
problems  stem  from  the  basic  difficulty  of  interpret¬ 
ing  program  goals  ....  Therefore,  very  fundamental 
changes  in  both  acquisition  procedures  and  management 
techniques  may  be  needed  if  actual  outcomes  are  to 
aporoach  oredicted  (estimated)  outcomes  more  closely 
[ 23: vii] .  * 


As  Carter  mentioned  earlier,  "Estimates  made  near 


the  beginning  of  a  development  program  are  particularly 
unreliable  [4:iii]."  Yet, 

The  major  commitment  to  these  expenditures  [referring 
to  life  cycle  expenditures  for  a  new  system]  is  made 
during  the  first  five  years  of  the  program  where  approx¬ 
imately  90%  of  costs  are  locked  in  ...  .  When  the 
product  is  finally  prototyped,  design  is  solidified  to 
a  point  where  less  than  25%  of  the  anticipated  PSD  [Full 
Scale  Development],  production  and  O&S  [Operating  &  Sup¬ 
port]  costs  can  be  influenced.  By  the  tine  production 
is  Initiated,  tooling  concepts,  manufacturing  and  facil¬ 
ities  plans  will  have  been  sufficiently  established  that 
major  changes  to  these  nonrecurring  costs  become  costly 
and  impractical  [2:90]. 

Therefore,  it  is  very  Important  that  correct  estimates  which 
take  into  account  uncertainty  and  risk  be  made  in  the  early 
phases  of  the  program,  preferably  the  conceptual  phase  since 
the  decisions  of  the  first  five  years  have  the  most  effect 
on  the  outcome  of  a  program. 

In  a  1981  thesis.  Second  Lieutenant  McClary  described 
three  areas  which  make  R&D  management  difficult. 

First,  upper  level  managers  sometimes  fail  to 
accurately  translate  their  understanding  of  corporate 
objectives  and  goals  into  the  technical  environment 
....  This  creates  the  problem  of  an  ill  defined  ■ 
objective.  .  .  .  Second,  because  most  of  the  sci¬ 
entists,  engineers,  and  technologists  in  the  R&D 
environment  are  moderately  knowledgeable  in  their  field 
of  discipline,  they  often  feel  that  they  should  decide 
which  projects  should  be  undertaken  by  the  organization 
....  Finally,  R&D  activities  are  different  when  com¬ 
pared  to  other  activities  of  business  such  as  production 
and  office  administration.  .  .  .  Probably  the  most 
Important  difference  is  that  It  is  difficult  to  measure 
outputs  of  R&D  activities  [20:1-2]. 

He  proposed  six  common  management  techniques,  which  he  chose 

after  a  literature  review,  to  be  presented  in  a  survey  of 


35 


R&D  managers  in  order  to  determine  which  techniques  were 
used  and  how  useful  they  were  in  R&D  management  (20:5-11). 

The  techniques  selected  were: 

Participative  Objective  Setting  Techniques  (POST), 
Project  Scheduling  Charts,  Work  Breakdown  Structures 
(WBS),  Activity  Networks,  Generalized  Activity  Net¬ 
works,  and  Cost  and  Schedule  Variance  Analysis  [20:5], 

The  value  of  the  above  techniques  in  risk  and  uncer¬ 
tainty  management  is  in  the  detail  the  manager  is  required 
to  use  in  their  application.  By  looking  at  the  detail 
involved  in  a  new  R&D  effort,  the  likely  areas  of  uncertainty 
and  risk  are  more  easily  identified. 

Lieutenant  McClary  found  that  "The  managers  were  most 
familiar  with  the  project  scheduling  charts  and  least  famil¬ 
iar  with  the  generalized  activity  networks  [20:62]." 

Project  scheduling  charts  include  the  Gantt  chart  and  the 
milestone  chart.  The  Gantt  chart  consists  of  a  bar  or  line 
chart  with  a  time  line  along  the  horizontal  axis  and  people, 
machines,  tasks,  etc.,  shown  as  the  variables  on  the  verti¬ 
cal  axis  (see  Figure  1).  The  horizontally  drawn  bars  show 
the  amount  of  time  a  particular  variable  should  use  (i.e., 
training,  ground  equipment,  etc.).  The  milestone  chart  is 
similar  to  the  Gantt  chart  but  only  identifies  the  timing  of 
particular  events  (see  Figure  2)  which  are  necessary  for 
project  success  (20:29).  Generalized  activity  networks 
include  such  computerized  solution  techniques  as  Venture 
Evaluation  and  Review  Technique (VERT) ,  Graphical  Evaluation 
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Figure  I.  Gantt  Chart.  Source:  AFSCP  800-3,  9  April  1976  [20:30] 
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Figure  2.  Milestone  Chart.  Source:  AFSCP  800-3,  9  April  1976  [20:31] 


and  Review  Technique  (GERT),  GERT  Simulation  (GERTS),  and 
Q-GERT  which  is  a  queueing  simulation  model  (20:40). 

He  also  found  that  Project  Scheduling  Charts  (PSC) 
(see  Figure  3)  were  used  more  frequently  than  the  other  six 
techniques  he  selected  to  be  included  in  his  survey  ques¬ 
tionnaire.  He  conducted  personal  interviews  with  R&D  man¬ 
agers  in  Aeronautical  Systems  Division  and  the  Aeronautical 
Laboratories  at  Wright-Patterson  Air  Force  Base  (AFB)(20: 
13-14,62).  Using  a  statistical  analysis,  he  found  that 

.  .  .  53%  of  the  managers  were  using  project  sched¬ 
uling  charts  alone  or  with  other  techniques  as  a  primary 
means  of  planning  their  project  activities.  ...  Of 
the  managers,  45?  used  PSC  or  some  combination  including 
PSC  as  a  primary  control  technique  [20:59]. 

In  a  telephone  conversation  with  Mr.  Jack  Gibson  (15) 
of  the  Air  Force  Systems  Command's  Aeronautical  Systems 
Division  (ASD)  at  Wright-Patterson  AFB,  it  was  found  that 
ASD  cost  analysts  use  various  cost  estimating  techniques 
depending  on  the  amount  of  time  they  have  to  complete  a 
study  and  the  complexity  of  the  project  being  estimated. 

They  have  a  library  of  estimates  of  completed  projects  from 
which  the  analyst  can  obtain  data  or  methods  of  estimating 
a  particular  system.  Different  techniques  were  used  within 
a  given  project,  depending  on  when  the  cost  estimate  was 
needed.  In  other  words,  the  amount  of  Information  available 
at  the  various  stages  of  the  project  determined  which  tech¬ 
nique  could  be  applied.  Mr.  Gibson  emphasized  that  there  is 
no  "cook  book"  approach  to  cost  estimating. 
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Figure  3.  Project  Scheduling  Chart.  Source:  AFSCP  800-3,  9  April  1976  [20:32] 


Captain  Scheel  found  that  "in  Air  Force  laboratories 
.  .  .  limited  cost  estimating  guidance  is  available  [31:92]." 
This  means  that  no  "cook  book"  was  reported  as  existing  by 

j 

the  forty- five  project  managers  he  personally  interviewed  in 
the  seven  R&D  laboratories  selected  for  analysis  (31:11). 

In  his  introduction  he  described  causes  of  risk  and  uncer¬ 
tainty  such  as  inflation,  requirement  changes,  schedule 
delays,  policy  changes,  and  the  fact  that  only  performance 
specifications  were  defined  at  the  beginning  of  an  R&D 
project  (31:1). 

Because  future  funding  requirements  for  R&D  pro¬ 
grams  are  largely  determined  by  the  initial  or  baseline 
cost  estimates,  it  is  critically  important  for  project 
managers  to  minimize  controllable  error  in  cost  esti¬ 
mating  [31:2]. 

Two  classes  of  commonly  used  estimating  techniques 
usable  when  project  cost  is  developed  are  as  follows: 

(1)  costing  by  analogy,  in  which  costs  of  similar 
systems  are  used  [as  a  basis]  for  the  estimate;  and 
(2)  cost  estimating  relationships  which  are  equations 
relating  to  system  performance  characteristics.  .  .  . 
These  techniques  do  not  guarantee  accuracy  because  of 
uncontrollable  factors  such  as  inflation  or  sudden 
price  increase  of  foreign  procurred  raw  materials 
[31:2]. 

Captain  Scheel  concluded  the  following  based  on  an 
analysis  of  the  thirty-five  questions  asked  in  his  interviews: 

...  in  an  Air  Force  laboratory  .  .  .  limited  cost 
estimating  guidance  is  available.  Project  managers  rely 
almost  exclusively  on  historical  data  from  past  work 
units  or  recent  cost  proposals  and/or  their  own  project 
management  experience  accumulated  over  the  years  [31:92]. 

The  project  manager  must  rely  on  his  own  experience 
or,  if  a  novice  project  manager,  the  guidance  of  seasoned 
project  managers  or  experienced  supervisors  [31:10], 
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.  .  .  Experienced  project  managers  generally  do  not 
have  a  specific  method  to  estimate  costs,  thus  it  would 
be  difficult  if  not  impossible  to  instruct  the  new 
project  managers  in  a  particular  technique  [31:94]. 

The  project  managers  attributed  the  following  as 
the  major  factors  which  contribute  to  inaccurate  esti¬ 
mates  : 


1.  Laboratory  project  managers  inaccurately  esti¬ 
mate  manpower  (both  manhours  and  manhour  rates). 

2.  Laboratory  project  managers  inaccurately  esti¬ 
mate  overhead  rates. 

3.  Laboratory  project  managers  are  required  to  use 
optimistic  inflation  factors  [this  was  not  supported  by 
Captain  Scheel's  research][31:98] . 

4.  Potential  offerors  misinterpret  the  Statement  of 
Work  (SOW)[31:97]. 

.  .  .  Many  project  managers  do  not  retain  their 
original  cost  estimate  [31:101].  [This  means  that  the 
manager  cannot  compare  the  final  cost  to  the  original 
in  order  to  determine  how  accurate  he  estimated  the  final 
cost .  ] 


.  .  .  The  Job  Order  Cost  Accounting  System  (JOCAS) 
is  not  a  useful  cost  estimating  tool  [31:101], 

.  .  .  Managers  felt  constrained  by  the  initial  cost 
estimate  made  during  the  planning  cycle,  or  were  con¬ 
strained  by  the  laboratory  or  division  budget.  These  con¬ 
straints  result  in  the  following  consequences: 

1.  The  scope  of  the  effort  is  reduced.  As  a  result, 
the  technical  objectives  of  the  work  unit  may  suffer. 

2.  The  project  is  stretched  in  duration.  As  a 
result,  the  total  dollar  value  of  the  project  may  increase 
and  new  projects  planned  for  the  outyears  may  be  deleted. 

3.  Planning  estimates  are  always  inflated  to  avoid 
an  underestimation  of  costs  [31:102]. 

Although  the  results  of  the  above  study  are  very 
interesting,  one  should  keep  in  mind  that  this  study  was  con¬ 
ducted  at  the  Air  Force  Laboratories  and  did  not  apply  to 
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the  product  divisions  such  as  Aeronautical  Systems  Division 
(ASD),  Armament  Division  (AD),  etc.  The  main  difference 
is  in  the  final  product  produced:  the  laboratories’  main 
output  is  new  technology  (i.e.,  paper  studies)  while  the 
product  divisions’  main  output  is  hardware. 

In  traditional  management  theory,  managers  are 
expected  to  collect  and  weigh  facts  and  probabilities, 
make  an  optimal  decision  and  see  that  it  is  carried 
out.  In  developmental  projects,  a  clear  sequence  is  not 
possible  because  of  the  extended  duration,  the  many 
technical  unknowns,  the  continual  discovery  of  new  "facts” 
and  the  changing  constraints  and  pressures.  The  process 
must  be  designed  that  allows  recommitment,  reassessment 
and  redirection  without  allowing  never-ending  improve¬ 
ment  and  excuses  for  missing  budgets  and  schedules  [34:15]. 

Risk  has  two  sides,  the  probability  of  failure  and 
the  consequences  of  failure.  Uncertainty  also  has  simi¬ 
lar  considerations.  ...  In  any  analysis,  as  the  risks 
and  uncertainties  are  identified,  alternated  approaches 
must  be  developed  and  the  cost,  schedule  and  performance 
Impacts  considered.  Contingencies  must  then  be  built 
into  the  acquisition  strategy  to  cover  the  anticipated 
cost  escalation,  correct  possible  performance  shortcom¬ 
ings,  and  provide  adequate  time  in  the  schedule  to  recover 
[34:17]. 

General  Thurman  challenged  the  attendees  at  a  con¬ 
ference  on  Management  of  Risk  and  Uncertainty  to  consider  the 
following: 


1.  ...  Specific  techniques  are  needed  to  address 
the  front  end  of  the  acquisition  where  data  is  lacking 
and  suspect.  More  needs  to  be  done  on  managing  uncer¬ 
tainty  based  on  data  developed  for  similar  systems. 

2.  ...  [one  should]  not  only  look  at  models  and 
their  interaction  with  the  data  for  "out  of  tolerance" 
conditions,  but  to  look  at  trends  In  the  data  to  antici¬ 
pate  "out  of  tolerance"  conditions  in  the  future.  Well- 
publicized  computerized  schedule  control  techniques, 
such  as  PERT  networks,  are  often  well  behind  the  action. 
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_  3*  ...  [The]  last  challenge  is  to  improve  communi¬ 
cations  to  both  the  professional  and  the  analyst's  pro¬ 
gram  manager  [34:21-24]. 

Before  concluding  this  section,  a  summary  of  the 
Defense  Advanced  Research  Projects  Agency  (DARPA)  analysis 
of  technical  risk  and  uncertainty  of  selected  DARPA  programs 
"during  the  period  of  FY  1977  through  FY  1980  [8:1]"  should 
be  considered.  The  objective  of  the  effort  was  to  establish 
a  data  base  containing  historical  cost  growth  information  of 
major  R&D  and  production  programs  (8:1).  The  report  included 
an  analysis  of  the  data  collected  and  "a  preliminary  assess¬ 
ment  of  the  implications  of  such  analysis  for  management 
reserves  for  related  high-risk  DARPA  programs  [8:1]." 

Based  on  statistical  analysis  of  these  data  subsets 
[included  programs  from  such  agencies  as  DOD,  NASA,  etc.], 
trends  relating  probabilistic  measures  of  cost  growth 
were  developed.  These  trends  displayed  a  remarkable 
consistency  for  all  data  sets.  .  .  .  The  analysis  shows 
that  there  is  a  predictable  relationship  between  the 
size  of  an  original  estimate  of  program  overrun  (i.e., 
management  reserve)  and  the  confidence  level  that 
eventual  cost  growth  will  remain  within  that  limit.  .  .  . 
At  the  95  percent  confidence  limit,  DARPA  cost  growth 
factors  (i.e.,  .the  ratio  of  actual  program  costs  to 
original  estimates)  lie  in  the  range  of  approximately 
40  to  110  percent.  The  analysis  also  permits  management 
to  estimate,  in  the  aggregate,  what  total  program  man¬ 
agement  reserves  should  be  as  a  function  of  confidence 
limit  [8:1-2]. 

DARPA  did  not  define  the  term  "management  reserve"  but  it 
appeared  that  they  were  referring  to  the  difference  between 
the  baseline  or  original  estimate  and  the  current  estimate 
as  found  in  the  Selected  Acquisition  Reports  (SAR).  It 
should  ce  kept  in  mind  that  there  is  a  need  for  further 


study  of  the  causal  nature  of  cost  growth  and  risk  as  well 
as  the  limitations  imposed  by  the  sloe  and  the  nature  of 
the  data  base  before  the  above  results  can  be  fully  used 
(8:2).  The  final  DARPA  report  (9)  proposed  a  model  based  on 
the  earlier  study  (8)  which  will  be  described  later  in  this 
chapter. 

This  section  has  covered  a  span  of  eighteen  years 
and  has  presented  some  views  on  the  problems  involved  in 
risk  and  uncertainty  management.  Some  factors  which  con¬ 
tribute  to  uncertainty  have  also  been  presented.  This 
review  represents  a  cross-section  of  the  many  articles  writ¬ 
ten  about  the  problems  concerning  the  management  of  risk  and 
uncertainty.  The  next  section  presents  a  summary  of  cost 
estimating  techniques  since  it  is  difficult  to  understand 
the  uncertainty  and  risk  assessment  techniques  without  a 
knowledge  of  the  errors  that  can  occur  when  current  cost 
estimating  techniques  are  used  to  predict  future  costs  in 
a  new  project  or  weapon  system.  It  is  followed  by  a  section 
on  uncertainty  and  risk  assessment  techniques. 

Cost  Estimating  Techniques 

This  section  will  introduce  several  commonly  used 
cost  estimating  techniques.  The  purpose  of  this  section  is 
not  to  provide  an  in-cepth  description  of  the  various  tech¬ 
niques,  but  to  provide  a  brief  introduction  to  these  cost 
estimating  techniques  as  a  frame  of  reference.  A  basic 


understanding  of  cost  estimating  techniques  is  necessary  in 
order  to  understand  uncertainty  and  risk  analysis  techniques. 
While  cost  estimating  techniques  try  to  incorporate  some  of 
the  unknowns  in  future  costs ,  they  do  not  normally  take  into 
account  the  probability  distributions  associated  with  all 
the  different  variables,  both  known  and  unknown.  As  the 
name  implies,  they  are,  at  best,  an  estimate.  The  less  risk 
and  uncertainty  Involved  in  the  project  being  estimated,  the 
more  likely  we  are  to  have  an  accurate  cost  estimate.  For 
example,  a  cost  estimate  for  a  current  model  of  an  F-4 
fighter  would  be  much  more  accurate  than  a  cost  estimate  for 
a  B-i  bomber,  which  is  currently  in  development. 

A  cost  estimate  is  a  judgment  or  opinion  regarding 
the  cost  of  an  object,  commodity,  or  service.  This 
judgment  or  opinion  may  be  arrived  at  formally  or  inform¬ 
ally  by  a  variety  of  methods,  all  of  which  are  based  on 
the  assumption  that  experience  is  a  reliable  guide  to 
the  future.  .  .  .  The  techniques  used  for  estimating 
hardware  cost  range  from  intuition  at  one  extreme  to  a 
detailed  application  of  labor  and  material  cost  standards 
at  the  other  [36:1-2]. 

Although  the  advantage  of  cost  estimating  techniques  lies  in 
the  fact  that  they  do  not  require  the  use  of  probability 
density  functions,  a  disadvantage  can  be  found  when  statis¬ 
tical  techniques  must  be  applied  via  computer  in  order  to 
analyze  a  large  data  base.  Cost  estimating  methods  or 
approaches  can  range  from  simple  rules  of  thumb  to  the  com¬ 
plexities  of  the  parametric  approach.  These  will  be  dis¬ 
cussed  in  the  following  paragraphs. 


Expert  Opinion.  Expert  opinion  or  round-table 
estimates  are  the  best  guesses  of  an  expert  in  the  field 
which  relates  to  the  system  being  estimated.  It  is  the  sim¬ 
plest,  cheapest  method.  It  can  also  be  used  when  there  is 
no  data  available.  The  main  disadvantage  is  that  the  esti¬ 
mate  is  difficult  to  verify. 

Analogy .  The  analyst  makes  use  of  limited  histori¬ 
cal  information.  "The  estimator  collects  resource  informa¬ 
tion  on  a  similar  or  analogous  task  and  compares  the  task  to 
be  estimated  with  the  similar  or  analogous  one  [32:47]." 

For  example,  if  the  estimator  finds  that  the  new  task  takes 
twice  as  much  material,  labor,  time,  etc.  as  the  analogous 
task,  he  will  multiply  the  known  cost  of  the  analogous  task 
by  a  factor  of  two.  This  will  be  the  estimated  cost  of  the 
new  task.  While  this  method  has  the  advantage  of  being 
simple  and  cheap,  it  is  possible  that  the  estimator  could 
fail  to  identify  subtle  differences  between  the  two  tasks 
and  thus  use  an  incorrect  adjustment  factor. 

Parametric  Approach.  The  parametric  approach  is  a 
method  which  involves  the  "collecting  and  organizing  [of] 
historical  information  through  mathematical  techniques  and 
relating  this  information  to  the  work  output  that  is  being 
estimated  [  32:52]."  "Many  estimating  relationships  are 
simple  statements  that  indicate  the  cost  of  a  commodity 
is  directly  proportional  to  the  weight,  area,  volume,  or 
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other  physical  characteristics  of  that  commodity  [29:33]." 
Statistical  analysis  can  be  used  to  derive  these  relation¬ 
ships.  Simple  linear  regression  and  curvilinear  analysis 
will  be  discussed  in  the  following  paragraphs. 

Simple  linear  regression  is  used  when  two  variables 
appear  to  be  logically  related  in  a  linear  form.  It  is  a 
simple  two  variable  equation  of  the  form  y  =  a  +  bx,  where 
"a"  is  a  constant  and  "b"  is  a  coefficient  of  "x"  which 
represents  one  of  the  variables.  "x"  could  represent  the 
weight  of  a  missile  while  "y"  could  represent  the  cost. 

The  values  of  "a"  and  "b"  are  determined  by  a  method  called 
least-squares.  Basically  the  least-squares  method  measures 
the  sum  of  the  error  squared  of  each  data  point  from  the 
line  descrioed  by  the  above  equation  and  through  the  simul¬ 
taneous  solution  of  two  equations  involving  "a"  and  "b"  and 
the  data,  the  coefficients  are  derived  which  best  fit  the 
equation  to  the  data,  resulting  in  the  least  value  of  the 
sum  of  the  squared  errors. 

Multiple  linear  regression  is  used  when  it  is  found 
that  more  than  one  variable  can  describe  the  cost  relation¬ 
ship  withir.  the  project.  For  a  three-variable  situation 
the  equation  has  the  form  y  =  a  +  bx  +  cz.  Again,  "a"  is 
a  constant  while  "b"  and  "c"  are  coefficients.  An  example: 

C  =  -3.752  +  . 104(w)  +  .ClS(F) 
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where 

C  s  coat  in  thousands  of  dollars 

W  *  weight  in  pounds 

F  =  frequency  in  megahertz 

[  35:66]  . 

The  process  of  deriving  the  coefficients  is  more  complicated 
but  can  be  found  in  any  good  statistics  book.  Also,  there 
are  computer  techniques  which  can  calculate  the  coefficients 
for  both  this  model  and  the  simple  linear  regression  model. 

Curvilinear  analysis  is  used  when  logic  and/or  anal¬ 
ysis  shows  that  the  linear  relationships  do  not  accurately 
model  the  relationship.  For  example,  the  equation  could  have 
the  forms:  y  =  a  +  bx2,  y  =  ax°,  or  y  =  a  +  b/x.  There  are 
many  other  types  of  non-linear  functions  which  can  also  be 
used.  Again,  statistical  packages  available  on  a  computer 
can  calculate  the  coefficients. 

As  with  any  statistical  technique,  certain  assumptions 
must  be  met  or  the  models  above  will  be  in  error.  The  con¬ 
ditions  are  as  follows: 

1.  The  x  values  are  nonrandom  (fixed)  variables. 

2.  The  residual  deviations  are  Independent  random 
variables  with  normal  distributions  [about  the  true 
regression  surface]. 

3.  The  expected  value  of  the  distribution  of  each 
of  these  random  variables  is  zero,  and  the  unknown 
variance  is  the  same  for  all  values  of  x  [29:^6], 

The  cost  estimating  relationships  (CEF.s)  are  best  used  by  a 

person  who  is  familiar  with  the  equipment  for  which  the 
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relation  was  developed.  CERs  are  only  useful  within  a  1 im- 
ited  range  (29:81-87). 


Industrial  Engineering  (Detailed)  Approach.  Another 
estimating  technique  is  the  industrial  engineering  approach. 
For  this  project,  the  project  is  broken  down  into  segments 
at  a  low  enough  level  of  detail  so  that  labor  and  material 
standards  can  be  applied  to  come  up  with  an  estimate  of  the 
project  cost  (sometimes  called  grass  roots  estimating) ( 29 : 
2-7).  Time  and  motion  studies  can  be  used  to  derive  the 
necessary  data  to  compute  the  standards  needed  for  an  esti¬ 
mate.  The  tecnnique  has  the  following  disadvantages  for 
government  use: 

It  is  difficult  to  establish  standards  that  will 
be  applicable  at  some  specified  production  quantity  in 
advance  of  production  experience. 

Industrial  engineering  estimating  procedures  requi 
considerably  more  personnel  and  data  than  are  likely  t 
be  available  to  government  agencies  under  any  foresee¬ 
able  conditions  [especially  early  in  the  system's  devel¬ 
opment]. 

They  take  too  much  time  and  are  costly  to  both  con¬ 
tractor  and  government  during  a  period  of  limited  funds. 
Moreover,  for  many  purposes  they  have  been  found  to  be 
less  accurate  than  estimates  made  statistically  [29:5]. 

Essentially  the  industrial  engineering  approach  col¬ 
lects  all  the  estimated  manhours  and  materials  of  each  ele¬ 
ment  and  combines  them  with  the  subelements  of  work.  The 
cost  of  the  elements  and  subelements  are  summed  to  arrive  at 
a  total  cost  estimate  (32:6). 

In  summary, 
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The  widespread  use  of  estimating  relationships  in 
the  form  of  simple  cost  factors,  equations,  curves, 
nomograms,  and  rules  of  thumb  attests  to  their  value 
and  to  the  variety  of  situations  in  which  they  can  be 
helpful.  But  an  estimating  relationship  can  only  be 
derived  from  information  on  past  occurrences,  and  the 
past  is  not  always  a  reliable  guide  to  the  future.  .  .  . 
estimating  relationships  .  .  .  are  an  important  tool  in 
an  estimator's  kit  and,  in  many  cases,  the  only  tool. 

...  The  limitations  of  estimating  relationships  stem 
from  two  sources:  first,  the  uncertainty  inherent  in 
any  application  of  statistics  and  second,  the  uncer¬ 
tainty  that  an  estimating  relationship  is  applicable 
to  a  particular  article.  The  first  pertains  primarily 
to  articles  well  within  the  bounds  of  the  sample  on 
which  the  relationship  is  based;  even  here,  uncertainty 
may  be  found.  The  second  source  refers  to  those  cases 
in  which  the  article  has  characteristics  somewhat  dif¬ 
ferent  from  those  of  the  sample  [29:79]. 

As  mentioned  at  the  beginning  of  this  section,  only 
a  brief  description  was  given  of  the  more  widely  used  cost 
estimating  techniques.  They  ranged  from  analogy  using  adjust 
ment  factors,  which  can  give  a  quick  overall  estimate  without 
complex  mathematical  approaches,  to  parametric  methods, 
which  can  involve  complex  mathematical  manipulations  of  his¬ 
torical  data.  While  these  techniques  can  be  used  to  obtain 
a  point  estimate  of  the  future  cost  of  a  new  system,  they 
were  not  designed  to  be  used  alone  to  provide  a  total  cost 
estimate  that  includes  risk  and  uncertainty.  Therefore,  the 
next  section  will  deal  with  techniques  that  can  be  used  to 
incorporate  risk  and  uncertainty  into  the  total  cost  esti- 
mat  e . 


Uncertaint1 


Risk  Techniques 


In  applying  the  techniques  described  in  this  section, 
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It  must  be  kept  in  mind  that  " .  .  .  it  is  extremely  diffi¬ 
cult  to  approach  the  field  of  risk  analysis  with  the  idea 
that  a  checklist  or  cook  book  type  methodology  provides  the 
answer  [ 18 : 3 8 3 • "  Cost,  schedule r  and  performance  have  been 
found  to  be  interrelated.  "These  interrelationships  are  not 
constant  throughout  the  system  life  cycle  [18:40]."  For 
example,  "Changing  performance  or  schedule  parameters  late  in 
the  life  cycle  do  not  produce  the  same  monetary  results  as  an 
equal  change  which  occurs  earlier  [18: 40]."  It  is  quite  dif¬ 
ficult  to  derive  mathematical  equations  which  adequately  des¬ 
cribe  all  the  interdependencies  involved  in  a  real  world  situ¬ 
ation  as  well  as  to  recognize  and  choose  the  ones  which  have 
the  most  significant  effect  on  the  R&D  project. 

"The  program  manager  is  attempting  to  buy  the  best 
possible  system  for  the  least  possible  cost  and  assure  its 
delivery  within  the  required  time  [l8:70]."  In  order  to  do 
this,  the  program  manager  must  apply  both  objective  and  sub¬ 
jective  techniques  so  that  a  balance  is  achieved  between 
"cost,  schedule,  and  performance  In  a  manner  which  best 
enables  him  to  fulfill  the  objectives  of  the  program  [18:70]." 
"Since  risk  is  dependent  both  on  the  amount  of  uncertainty 
involved  and  on  the  management  decisions  made  in  the  procure¬ 
ment  process  [18:9],"  "any  risk  assessment  program  must  be 
tailored  not  only  to  the  situation  being  assessed  but  to  the 
individuals  who  must  perform  the  assessment  [18:84]." 

Objective  techniques  are  based  on  measurements  of 
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directly  observable  phenomenon  such  as  a  statistical  esti¬ 
mate  based  on  earlier  collected  data.  Objective  techniques 
have  "proved  highly  successful  in  many  historical  situations 
such  as  the  management  of  gambling  houses,  industrial  quality 
control  and  numerous  scientific  endeavors  [18:^8]."  It  must 
be  remembered  that  many  statistical  techniques  have  been  best 
applied  to  repetitive  phenomenon  ( 18 : 48 ) . 

Subjective  techniques  are  best  used  where  nonrepeti- 
tive  phenomenon  occur.  The  weapon  systems  process  is  a  case 
in  point.  It  is  not  a  stable,  repetitive 

process  and  the  use  of  individual  subjective  opin¬ 
ions  must  be  used  to  acquire  the  estimates  of  the 
future  events  either  in  the  form  of  probability  density 
functions  or  other  types  of  estimates  [18:49]. 

The  subjective  opinions  are  based  on  past  experience. 

Because  the  analysis  of  risk  and  uncertainty  must  start  at 

the  early  phase  of  a  program,  probability  theory  must  be 

applied  carefully  since  the  "vast  amount  of  historical  data 

available  concerning  past  system  acquisition  .  .  .  [is]  not 

[a]  stable  repetitive  process  as  defined  by  probability 

theory  [ 18:127]." 

When  a  future  program  is  evaluated  similar  programs 
or  systems  from  the  past  are  used  to  develop  an  estimate  of 
the  present  program’s  cost,  schedule,  and  performance  param¬ 
eters.  This  amounts  to  inferring  one  population's  param¬ 
eters  from  another  population’s  data  which  is  statistically 
incorrect.  Therefore,  ’’the  analysis  of  the  risk  involved 
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in  advanced  systems  must  involve  the  use  of  probabilities 
which  cannot  be  based  on  direct  experimentation  [18:46]." 
Since  "risk  assessment  of  a  weapon  system  acquisition  pro¬ 
gram  is  time  dependent  and  a  function  of  the  specific  phases 
of  the  program  life  cycle  [18:36],"  different  types  of  risk 
analysis  must  be  performed  at  the  different  stages  of  the 
system's  life  cycle  ( 18:42). 

The  Bayesian  Approach.  Further  work  was  accomplished 
on  El-Sabban's  Bayesian  model  by  Major  Lenox  in  1973.  The 
proposed  Bayesian  approach  incorporated  present  knowledge 
of  a  project's  status  with  a  knowledge  of  the  past  histori¬ 
cal  systems  performance  data  to  predict  the  future  values  of 
a  chosen  system  parameter  (i.e.,  cost,  schedule,  etc.). 

This  model  allowed  the  analyst  to  incorporate  a  weighting 
factor  for  his  subjective  feelings  of  confidence  concerning 
the  value  of  the  present  data  compared  to  the  value  of  the 
past  available  data.  The  model  allows  the  analyst  to  evalu¬ 
ate  the  effect  of  various  weighting  factors  in  order  to  con¬ 
duct  a  sensitivity  analysis.  This  allows  him  to  choose  the 
best  weighting  factor  based  on  his  experience  (18:55). 

The  analyst  selects  a  distribution  which  represents 
the  probability  of  an  event  based  on  the  past/or  present 
state  of  knowledge  or  judgement.  This  probability  distribu¬ 
tion  is  selected  based  on  the  knowledge  of  the  mean,  vari¬ 
ance,  and  skewedness  of  the  historical  data  chosen  as  a 
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basis  for  estimating  the  costs  of  a  new  system.  Two  types 
of  distributions  commonly  used  are  the  normal  distribution 
and  the  beta  distribution  (18:28-50). 

Normal  and  Beta  Probability  Distributions.  The 
normal  probability  distribution  is  the  familiar  bell-shaped 
curve  in  which  the  variable  of  interest  can  assume  any  value 
from  minus  infinity  to  plus  infinity.  The  distribution 
represents  the  probability  that  the  true  value  of  the  vari¬ 
able  being  measured  is  different  than  the  mean  which  is 
centered  at  the  highest  point  of  the  bell  curve.  "...  The 
probability  of  the  true  value  exceeding  the  mean  is  equal 
to  the  probability  of  the  mean  exceeding  the  true  value.  .  . 
[18:29]."  This  distribution  does  not  hold  true  in  all  situ¬ 
ations  since  "the  cost  often  has  a  greater  probability  of 
exceeding  the  mean  than  of  being  less  [18:29]."  This  is 
especially  true  when  considering  R&D  cost  situations. 

In  R&D  cost  situations,  the  beta  curve  is  the  better 
choice  many  times  and  has  found  wide  use  in  many  risk  tech¬ 
niques.  The  beta  distribution  looks  like  a  bell  curve  but 
it  does  not  have  to  be  a  symmetrical  bell  curve,  depending 
on  the  parameter  values  specified.  Three  parameters  must 
be  specified  in  order  to  derive  the  curve:  the  variable 
being  measured,  the  right  endpoint,  and  the  left  endpoint. 
The  right  and  left  endpoints  must  both  be  greater  than  or 
equal  to  zero  and  less  than  or  equal  to  one.  Although  the 
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beta  distribution  endpoints  are  restricted  to  a  range  of 
zero  to  one,  the  variable  of  interest  can  have  its  values 
transformed  so  that  they  will  fit  into  this  interval  (18: 
28-31).  Mote  that  the  normal  distribution  had  plus  or 
minus  infinity  for  endpoints. 

By  varying  the  "a"  and  "b"  values,  the  individual 
can  indicate  his  degree  of  confidence  in  the  parameter 
he  is  estimating  and  can  establish  the  shape  of  the 
curve  [  18: 31-32] [see  Figure  4]. 

The  beta  distribution  can  be  applied  to  a  wide 
variety  of  shapes.  By  using  the  three  parameters  to  repre¬ 
sent  the  following:  a  pessimistic  estimate,  an  optimistic 
estimate,  and  a  most  likely  value,  the  analyst  can  indicate 
any  degree  of  confidence  he  may  choose  (18:30-32).  ’’The 
beta  distribution  is  widely  used  in  analysis  due  to  its 
great  flexibility  [ 18: 30] "[see  Figure  5 ]. 

Networking  Techniques.  As  mentioned  earlier,  cost, 
schedule,  and  performance  parameters  are  interrelated  and 
it  is  difficult  to  quantify  these  interrelationships  mathe¬ 
matically.  The  use  of  network  techniques  provides  a  quali¬ 
tative  method  which  simultaneously  evaluates  these  param¬ 
eters  by  the  use  of  a  computer  C 18 : 7 1 ) -  The  various  net¬ 
working  techniques  follow. 

Program  Evaluation  and  Review  Technique  (PERT), 
developed  in  1958  for  the  Polaris  Program,  was  developed 
to  model  a  complex  project  involving  several  interrelated 
activities  (18:74).  The  project  to  be  modeled  is  broken  up 
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into  several  events  which  are  then  listed  in  sequential 
order  for  completion.  The  time  for  completion  for  each 
event  is  then  entered  into  the  model  along  with  the  correct 
order  for  each  event.  The  PERT  network  is  analyzed  and  the 
path  for  earliest  project  completion  (critical  path)  is 
found.  Any  changes  in  the  project  schedule  which  would  tend 
to  extend  this  time  would  require  management  attention  (IS: 
74-78). 

The  original  PERT  model  made  the  following  assump¬ 
tions  : 


1.  The  project  could  be  described  by  a  network  of 
activities  with  well-defined  precedence  relationships. 

2.  The  distribution  of  the  activity  time  for  each 
activity  could  be  described  by  a  Beta  distribution. 

3.  The  parameters  of  the  Beta  distribution  could 
be  estimated  from  three  subjective  estimates: 

a.  The  optimistic  (a) 

b.  The  pessimistic  (b) 

c.  The  most  likely  (m) 

4.  The  mean  of  the  distribution  was  approximated 

by  .  .  .  1/6  (a  +  4m  +  b)  and  the  standard  deviation  was 
approximated  by  .  .  .  1/6  (b  -  a). 

5.  The  activity  times  were  independent  [18:76]. 

Some  of  the  shortcomings  of  PERT  are: 

1.  The  requirements  for  the  activity  times  to  be 
independent.  'This  shortcoming  exists  to  some  extent 
in  all  network  techniques. 

2.  Every  activit"  leading  to  an  event  must  be  com¬ 
pleted  before  that  event  can  be  realized.  Neither  can 
any  activity  following  that  event  be  initiated  until 
the  event  itself  has  been  realized.  This  is  a 
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significant  shortcoming  in  many  types  of  projects 
especially  those  involving  complex  systems  and  subsys¬ 
tems. 

3.  The  events  must  be  completed  in  a  particular 
sequence  in  order  to  complete  the  project.  This  may 
not  be  a  significant  shortcoming  in  most  projects  as 
they  involve  a  naturally  ordered  sequence  of  events 
[18:771. 

PERT  provides  a  limited  capability  for  the  use  of 
probabilistic  logic  but  would  be  "unrealistic  for  analyzing 
most  system  acquisition  programs  since  the  events  are 
assumed  to  be  deterministic.  .  .  [ 18:78].”  Graphical 
Evaluation  and  Review  Technique  (GERT)  was  developed  to  over¬ 
come  this  problem.  ”GERT  allows  an  event  to  be  realized 
without  completing  all  activities  leading  up  to  that  event 
[ 18:78]."  One  shortcoming,  though,  is  GERT's  "lack  of 
realism  in  handling  the  interdependencies  inherent  in  the 
acquisition  process  [18:78]." 

So  far  no  mention  has  been  made  of  incorporating 
probability  distributions  since  PERT  and  GERT  do  not  have 
this  capability.  '  Risk  Information  System  and  Cost  Analysis 
(RISCA)  was  one  of  the  techniques  developed  to  overcome 
this  problem. 

It  employs  the  use  of  simulation  and  Monte  Carlo 
procedures  [discussed  in  Chapter  4]  to  generate  an 
output  consisting  of  a  frequency  distribution  for  all 
possible  terminal  events  and  the  corresponding  time 
and/or  cost  distribution  for  each  terminal  event 
[ 18:79]. 

RISCA  is  much  easier  to  use  than  the  following  described 
Venture  Evaluation  and  Review  Technique  (VERT).  "VERT, 
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with  its  greater  flexibility,  is  more  difficult  to  use  and 
requires  substantially  more  memory  [3^:177]." 

Although  RISCA  overcame  some  of  the  problems  involved 
in  the  PERT  and  GERT  techniques,  it  still  has  some  short¬ 
comings.  It  is  not  able  to  "realistically  [deal]  with  the 
interdependencies  [i.e.,  cost  versus  time]  of  the  acquisi¬ 
tion  process  [18:80]."  While  it  can  handle  uncertainty 
fairly  well,  "RISCA  does  not  provide  for  the  risk  assessment 
of  the  performance  parameter  [18:81];"  it  will  analyze  the 
performance  parameter. 

Venture  Evaluation  and  Review  Technique  (VERT)  per¬ 
forms  better  than  RISCA  since  it  does  have  a  limited  capa¬ 
bility  to  handle  a  risk  assessment  of  performance  (18:83)* 
VERT  uses  network  simulation  and  Monte  Carlo  techniques 
which  result  in  the  computerized  output  as  follows: 

1.  Pictorial  histogram  approximations  to  the  prob¬ 
ability  density  function  and  to  the  cumulative  density 
function. 

2.  The  mean  observation,  indicating  the  center  of 
the  distribution. 

3.  The  standard  deviation,  indicating  the  disper¬ 
sion  of  the  distribution. 

4.  The  coefficient  of  variation  which  enables  the 
analyst  to  draw  inferences  concerning  the  spread  of  the 
distribution  in  relation  to  its  mean  [l3:8l]. 

VERT  has  the  ability  to  perform  a  "formalized  decision  risk 

analysis  [18:81]."  It  also  has  the  capability  of  applying 

cash  discount  formulas  (18:62). 
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VERT  has  14  transformations  of  relationship  to  aid 
(in'  the  task  of  expressing  the  functional  relation¬ 
ships  among  the  parameters  of  time,  cost,  and  perfor¬ 
mance.  .  .  .  VERT  has  sufficient  flexibility  to.  incor¬ 
porate  data  as  it  becomes  available  for  updating.  .  .  . 

VERT  enables  the  identif i-cat ion  of  the  following 
program  areas  or  items  for  consideration: 

1.  Potential  problems 

2.  Consequences  of  element  failure 

3.  Both  low  and  high  risk  program  areas 

4.  Requirements  versus  state-of-the-art  trade-offs 

5.  Adequacy  of  acquisition  time 

6.  Sufficiency  of  appropriations 

7.  Insufficient  performance 

8.  Optimum  allocation  of  funds,  time,  and  design 
effort 

9.  The  existence  of  data  gaps  i^hich  might  require 
further  study  or  experimentation 

10.  Sensitive  or  critical  parameters  [18:82-83]. 

One  disadvantage  of  VERT  is  the  level  of  expertise  required 
to  model  the  complex  interaction  of  all  the  variables  in  an 
R&D  program  (18:84). 

Computer  Models.  Some  other  models  which  are  avail¬ 
able  for  handling  the  interdependencies  of  the  acquisition 
process  are:  the  Systems  Engineering  Model,  the  Generalized 
Risk  Analysis  Model  (GRAM),  and  the  Multiple  Incentive  Con¬ 
tract  Evaluation  Model  (MICE).  These  models  are  used  to 
analyze  sub-areas  of  a  network  and  "they  may  grant  more 
accurate  analysis  of  a  specific  problem  area  within  the 
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network  than  a  general  type  network  analysis  [13:84]." 

The  Systems  Engineering  Model  is  an  integrated 
aircraft  evaluation  model  with  the  capability  for  sub¬ 
system  level  trade-offs.  ...  It  gives  an  insight 
into  a  specific  problem  area  in  greater  depth  than  a 
general  overall  analysis  of  the  complete  system  would. 

In  this  respect,  an  analysis  of  this  type  is  a  valu¬ 
able  input  to  alleviating  uncertainty  and  possibly 
reducing  risk.  Another  valuable  aspect  of  a  model  of 
this  type  is  that  it  has  been  specifically  designed 
to  handle  the  performance  parameter  [18:85-86]. 

GRAM  is  essentially  an  analysis  aid  "which  facili¬ 
tates  the  adaptation  of  an  evaluation  model  to  a  risk  anal¬ 
ysis  by  putting  it  into  a  Monte  Carlo  simulation  framework 
[29:86]."  MICE  is  a  technique  which  allows  a  defense  con¬ 
tractor  to  choose  among  several  alternatives  to  select  opti¬ 
mum  contract  parameters.  It  analyzes  each  alternative  or 
strategy's  effect  on  program  cost,  performance,  and  schedule 
( 18:87) . 

Before  concluding  this  section,  a  recent  article  by 
Dr.  McNIchols,  of  Management  Consulting  &  Research,  Inc. 
(MCR),  described  a  procedure  that  had  the  possibilities  of 
computerization.  Although  he  did  not  describe  the  mathe¬ 
matics  Involved,  he  proposed  a  procedure  for  cost  risk  assess 
ment  of  the  various  proposals  that  contractors  submit  to  meet 
government  specifications  in  a  particular  project.  The  pro¬ 
cedure  considers  these  risk  factors:  risk  assessment,  risk 
management ,  and  risk  demonstration,  while  taking  into  con¬ 
sideration  the  fact  that  "The  military  worth  of  a  weapon  sys 
tern  is  often  evaluated  by  several  criteria:  system 
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-performance,  life  cycle  cost  and  developmental  risk  [  34:249]." 

The  above  risk  factors  are  defined  as  follows : 

•  -  Risk  Assessment:  Identify  the  degree  of  techni¬ 

cal  risk  with  respect  to  realism,  soundness,  and 
credibility  (i.e.,  the  technical  feasibility  as  well 
as  the  "contractor  bias  problem"). 

-  Risk  Management:  Develop  a  plan  for  managing  all 
types  of  risk  (risk  minimization  plan)  as  a  function  of 
time  (i.e..  Acquisition  Milestone  I,  II,  and  III). 

-  Risk  Demonstration :  A  test  and  evaluation  demon¬ 
stration  plan  should  allow  early  identification  of  risks 

-  and  the  steps  required  to  reduce  high  risk  program  ele¬ 
ments  to  demonstrated  acceptable  levels  as  well  as  the 
cost  of  doing  so  [  34:249]. 

The  steps  involve  (1)  the  identification  of  the  key 
risks  that  are  relevant  to  a  particular  component  or  system 
as  well  as  the  technological  risk  involved;  (2)  the  develop¬ 
ment  of  measurement  standards  (i.e.,  cost)  as  a  measure  of 
effectiveness;  and  (3)  develop  an  analytic  methodology  that 
combines  the  several  types  of  risk  identified  as  a  means  of 
deriving  a  cost-risk  assessment  (34:250).  The  advantages 
stated  were: 

-  Consistency  across  technologies /manufacturers/ 
program,  phases. 

-  Ease  of  application  and  explanation. 

-  Ability  to  quantify  subjective  knowledge  of 
technical  experts  in  each  key  area  [34:250]. 

The  disadvantage  involved  In  the  MCR  procedure  concerns  the 

need  to  derive  a  probability  density  function  of  cost  for 

each  contractor's  design  and  a  composite  probability  density 

function  for  each  of  the  parallel  development  alternatives 
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(  34:251).  The  problem  lies  in  the  many  alternative  designs 
that  each  contractor  may  be  considering,  although  he  submits 
only  one.  This  results  in  comparing  contractors  based  on 
only  one  point  of  the  cost-risk  profile  without  knowing  the 
cost-risk  variation  from  the  contractor’s  probability  density 
function  (34:253).  It  is  likely  that  the  program  will  not 
have  enough  data  to  derive  the  contractor’s  probability 
density  function. 

This  subsection  has  described  some  of  the  computer 
models  which  have  been  developed.  They  basically  described 
a  static  situation  and  were  used  to  evaluate  only  a  part  of 
the  whole  program.  The  next  subsection  discusses  simulation 
models  which  are  able  to  look  at  the  dynamic  interactions 
of'  a  total  program. 

Simulation  Models.  In  1978,  E.  B.  Cochran  described 
a  model  which  could  be  used  to  measure  and  predict  disrup¬ 
tion  which  is  a  product  of  two  factors:  (1)  the  Inherent 
uncertainty  of  development  work  and  (2)  such  work  Is  per¬ 
formed  In  an  environment  of  urgency  in  order  to  obtain  fast 
delivery  to  the  military  services  (2:67).  The  model  he 
developed  is  "an  interactive  computer  program  which  allows 
the  user  to  enter  appropriate  design,  production  and 
schedule  characteristics  [2:71]."  Inputs  to  the  program 
include:  the  parameters  of  the  production  facility  (man¬ 

power,  storage  space,  capacity,  etc.),  work  center  costs. 
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the  master  schedule,  data  on  direct  effects  of  design  uncer¬ 
tainty,  and  other  factors  such  as  cost  penal-ty  functions 
(2:71).  The  program  uses  a  networking  technique  such  as 
PERT  for  part  of  its  data  manipulations. 

The  output  of  the  program  is  time-phased  comparisons 
between  projected  manhours  and  budgeted  (or  actual)  manhours. 
It  would  appear  that  this  is  the  same  as  a  variance  analy¬ 
sis.  The  model  allows  the  analyst  to  vary  the  model  param¬ 
eters  in  order  to  examine  their  effects.  Ke  may  also  try 
different  types  of  cost  functions.  The  model  had  good 
results  in  the  earlier  months  of  a  test  run  but  lost  some 
accuracy  due  to  weak  (In  the  author's  words)  data  on  changes 
and  delays  (2:74) . 

A.  J.  Rowe  and  I.  A.  Somers  defined  disruption  as 
the  disorder  and  turmoil  created  by  forces  which  were  not 
part  of  the  normal  program  plans  and  related  production 
procedures.  These  plans  and  procedures  are  normally  used 
to  minimize  program  costs  (3^:28).  Normally,  a  program  is 
planned  to  make  the  best  use  of  available  resources.  Pro¬ 
duction  procedures  developed  over  a  period  of  time  are  used 
to  keep  costs  to  a  minimum.  Any  unplanned  changes  In  these 
procedures  from  outside  sources  (i.e.,  strikes,  changes  in 
technology,  etc.)  cause  an  increase  of  costs  due  to  disrup¬ 
tion.  They  defined  "disruption  cost"  as  the  difference 
between  the  actual  program  cost  and  the  amount  that  would 
be  "reasonably  required"  without  the  penalties  for  late 
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incorporation  of  changes  needed  to  complete  the  project  for 
delivery  (34:28). 

They  proposed  a  Causal  Integrative  Model  (CIM)  to 
explain  the  disruption  pattern  found  in  any  project.  It  is 
an  extension  of  the  Forrester  type  dynamic  system  model 
which  is  a  dynamic  computer  model  with  the  following  features 

-  Several  levels  or  accumulations  within  the  system; 
e.g.,  inventories,  number  of  employees,  work  in  process. 

-  Flow  rates  that  transport  the  contents  of  one 
level  to  another. 

-  Decision  functions  that  control  the  rates  of  flow 
between  levels. 

-  Information  channels  that  connect  the  decision 
function  to  the  levels  concerned  [34:57]. 

The  Forrester  dynamic  system  model  has  the  advantage 
of  depicting  a  clear  picture  of  the  most  likely  areas  of  dis¬ 
ruption  problems.  It  also  forces  the  program  managers  to 
understand  their  decision-making  processes.  This  can  lead 
to  a  greater  Insight  into  the  program’s  problems  including 
their  cause  and  effect  relationships.  The  disadvantages 
include : 


-  In  simulation,  all  relevant  variables  and  phenomena 
must  be  quantified.  The  reduction  of  all  descriptive 
knowledge  to  quantitative  measures  is  not  always  valid. 

-  Dynamic  simulation  is  found  to  be  most  useful  in 
price-quantity  problems  [note  that  this  is  a  primary  DOD 
concern],  less  useful  in  organization  design  and  least 
useful  in  product-market  strategy. 

-  Dynamic  simulation  is  not  easy  to  apply.  It  is  a 
complex  technique  that  needs  considerable  data  and  know¬ 
ledgeable  people. 
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-  There  are  problems  in  acceptance  of  the  approach 
because  it  is  often  considered  a  research  tool  l3^*'57j. 

CIM  is  an  extension  of  the  Forrester  dynamic  system 


model . 


[It]  describes  the  processes,  flows,  variables, 
feedback  loops,  delays,  exogenous  variables  and  key 
decisions  as  they  are  related  to  the  four  basic  vari¬ 
ables  in  the  acquisition  process  [Internal  Control, 
Technological  Uncertainty,  Customer  Uncertainty,  and 
Environmental  Uncertainty ][ 3^: 57] . 

[The  CIM  model  includes].  .  .  many  causal  sub¬ 
models,  such  as  concurrency,  learning  curve,  disrup¬ 
tion,  etc.  It  also  covers  the  dynamic  interdependen¬ 
cies  that  exist  and  the  treatment  of  risk  and  uncer¬ 
tainty  as  integral  parts  of  the  model  [3^:58]. 


DARPA  Model.  In  the  section  on  uncertainty  and  risk 
management,  the  DARPA  Model  was  referenced  but  not  discussed. 
Historical  data  from  DOD,  Department  of  Energy  (DOE),  and 
NASA  was  used  to  build  a  data  base  to  represent  the  govern¬ 
ment's  acquisition  history.  Various  statistical  techniques 
were  used  to  obtain  trends  from  which  to  base  an  estimate  of 
future  overruns.  "A  'cost-growth  risk'  relationship  has  been 
developed  which  provides  a  relative,  but  quantitative,  mea¬ 
sure  of  risk  based  on  historical  experience  of  major  high 
technology  programs  .  .  .  [9:11-2].'' 

Overrun  magnitude  was  represented  in  dimensionless 
"cost  factor"  form,  that  is  the  ratio  of  the  year-end 
estimate,  or  actual  cost,  to  the  original  program 
baseline  cost  prediction  for  the  year  [9:11-31* 

They  found  that  for  a  cost  overrun  that  was  predicted 
to  lie  at  the  mean  value,  "one  could  be  97*5#  certain  that 
the  actual  overrun  would  not  exceed  the  mean  value  plus 


67 


twice  the  standard  deviation  [9:11-4]."  Experimental  Evalua¬ 
tion  of  Major  Innovative  Technologies  (EEMIT)  data  was 
plotted  using  a  ratio  of  actual  overruns  to  predicted  over¬ 
runs  to  derive  a  dimensionless  cost  factor.  The  results 
showed  a  bias  towards  developmental  programs  (9:11-4).  In 
other  words,  since  developmental  programs  have  higher  risk 
and  uncertainty,  the  final  costs  (actual)  will  be  higher  than 
originally  estimated.  The  bias  mentioned  above  refers  to 
the  fact  that  the  developmental  programs  have  larger  cost 
factors  than  the  more  mature  programs.  The  plotted  data  is 
used  to  derive  "zero  risk  cost  growth"  factors  in  order  to 
derive  a  budget  that  represents  a  1005?  confidence  of  zero 
cost  growth  (9:11-6). 

The  result  of  this  DARPA  effort  is  a  proposal  that 
by  the  use  of  this  data  base,  upper  management  would  provide 
top-down  risk  management.  They  would  calculate  what  each 
program  would  receive  as  a  management  reserve  for  risk  by 
the  use  of 

distribution  factors  based  on  program  funding 
profiles.  .  .  .  [and]  influence  factors  based  on  non¬ 
specific  program  characteristics  .  .  .  determined  by 
consensus  review  of  competing  EEMIT  programs  by  six 
DARPA  "technology"  offices  [9:11-7]. 

"The  proposed  system  will  manage  only  risk  and  will  not  pre¬ 
empt  the  program  managers’  functions  [9:111-2]." 

It  is  the  researcher ' s  opinion  that  this  is  a  move 
by  DARPA  to  create  a  centrally  controlled  "risk  fund"  out¬ 
side  the  control  of  the  program  managers.  DARPA  would 
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apparently  manage  this  fund  and  determine  how  much  risk 
money  would  be  assigned  to  each  project.  Essentially,  this 
concept  of  centralized  management  would  preempt  some  of  the 
manager's  control  over  his  project  by  requiring  him  to 
obtain  higher  level  management's  permission  to  expend  the 
risk  money  to  cover  what  the  manager  perceives  as  cost  due 
to  risk.  The  manager  could  perceive  that  he  would  appear 
incompetent  if  he  applied  for  some  of  the  risk  money.  There¬ 
fore,  he  might  build  in  a  hidden  reserve  to  avoid  unnecessary 
exposure  to  the  higher  level  management . 

Subjective  Methods.  The  major  problem  with  any  of 
the  risk  techniques  has  been  how  to  determine  the  probability 
distributions  with  the  given  amount  of  data  available.  It 
is  difficult  for  the  analyst  to  specify  the  appropriate  sub¬ 
jective  probabilities.  These  subjective  probabilities  are 
especially  needed  when  there  is  little  relative  data  to  be 
found  concerning  a  new  project.  Major  Lenox  summarized  eight 
methods  to  derive  a  subjective  probability  density  function 
and  provided  references  for  the  interested  reader  (18).  They 
were  identified  as: 

1.  Choice-Between-Gambles  Technique 

2.  Choice-Between-Gambles  Technique  Deriving 
Cumulative  Distribution  Functions 

3.  Standard  Lottery 

4.  Modified-Churchman-Ackoff  Technique 

5.  Modified  Delphi  Technique 
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6.  Direct  Approximation  of  Probabilities 

7.  Fit  Values  to  Given  Shapes  of  Probability 
Functions 

8.  Fit  Curves  from  Point  Estimates  [18:131-132]. 
Colonel  Martin  also  provides  some  further  information  con¬ 
cerning  the  use  of  these  techniques  in  his  article  (34:182). 

Quite  often,  analysts  must  develop  subjective  prob¬ 
abilities  in  order  to  derive  an  estimate  of  uncertainty 
which  can  be  represented  as  a  risk  value.  Yet,  many  analysts 
have  not  had  formal  training  in  this  area.  Moskowitz  and 
Sarin  conducted  some  experiments  in  order  to  determine  whether 
"a  high  frequency  of  implicit  violations  of  the  probability 
calculus  were  observed  [34:236]."  They  were  endeavoring  to 
evaluate  the  use  of  a  Joint  Probability  Table  (JPT)  as  an 
aid  to  selecting  conditional  probabilities  in  decisions. 

They  noted  that  joint  probabilities  are  an  important  input  to 
decision  models  (34:236).  They  concluded  that: 

conditional  probabilities  are,  however,  basic  input 
to  many  forecasting  and  decision  models.  We  have  shown 
that  a  large  number  of  consistency  violations  of  the 
probability  calculus  occur  in  a  direct  assessment  of 
conditional  probabilities.  These  violations  .  .  . 

.  result  from  systematic  perceptual  and  cognitive  biases 
in  subjects’  responses.  ...  In  the  interim  we  pro¬ 
pose  that  an  analyst  should  make  the  decision  maker 
aware  of  the  consistency  conditions  and  discuss  with 
him  qualitatively  the  impact  of  evidence  on  the  event 
for  which  conditional  probability  is  being  assessed. 

The  analyst  should  also  use  an  assessment  aid  such  as 
JPT  in  the  elicitation  process  [34:242-243]. 

It  has  been  seen  so  far  that  not  only  is  the  subjec¬ 
tive  probability  difficult  to  measure,  but  that  many  analysts 
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may  have  problems  In  correctly  Identifying  the  subjective 
conditional  probabilities  involved  in  a  decision.  Further 
research  was  reported  by  Yager  concerning  the  application  of 
fuzzy  subsets  as  a  means  of  determining  how  to  model  a  per¬ 
son’s  approximate  reasoning  (34:185-187).  "Fuzzy  set  theory 
provides  a  framework  in  which  to  construct  models  for  multi¬ 
ple  objective  decision  making  [34:189]."  What  all  this 
means  is  that  a  method  is  being  researched  that  would  advance 
"the  idea  of  approximate  reasoning  as  a  means  of  simulating 
a  large  class  of  human  reasoning  operations  [34:185]."  This 
could  lead  to  another  method  of  modeling  the  subjectivity 
involved  in  the  acquisition  process.  The  mathematics 
involved  in  the  set  theory  is  used  to  analyze  a  new  form  of 
uncertainty  called  possibilistic  uncertainty.  Possibilistic 
uncertainty  answers  the  question  of  "how  easy  it  is  for  a 
particular  outcome  to  occur  in  a  given  time  [34:185-189]." 
Approximate  reasoning  is  able  to  take  imprecise  values  for 
some  variables  (i.e.,  x  =  about  45)  and  combine  them  with 
statements  describing  a  relationship  between  variables  (i.e, 
if  x  is  near  50,  then  y  is  much  greater  than  100)  to  derive 
possibility  distributions  (34:187).  This  implies  that  impre¬ 
cise  statements  could  be  given  a  quantifiable  measure. 

Summary .  Before  1970,  there  was  little  done  in  the 
area  of  risk  and  uncertainty  management  and  analysis.  Since 
then  there  has  been  a  growing  interest  with  an  ever 
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increasing  amount  of  published  literature.  Many  techniques 
have  been  developed  for  both  cost  estimation  and  risk  analy¬ 
sis.  While  the  actual  management  of  risk  and  uncertainty 
has  been  mentioned  in  most  articles,  very  few  have  addressed 
in  depth  the  needed  management  policies  to  assure  the  success 
of  a  risk  management  program.  Most  of  the  articles  have 
addressed  the  need  for  risk  management  and  described  the  fac¬ 
tors  which  influence  risk,  uncertainty,  and  cost  growth.  A 
few  of  the  more  recent  articles  have  addressed  risk  manage¬ 
ment  in  greater  depth  which  would  indicate  an  increasing 
interest  in  the  actual  management  of  risk  and  uncertainty. 

The  various  cost  estimating  techniques  mentioned 
earlier  provide  an  estimate  of  the  total  future  costs  of  a 
program  based  on  an  evaluation  of  past  data  on  similar  sys¬ 
tems  or  subsystems.  While  these  cost  estimates  do  not 
identify  risk  and  uncertainty,  they  do  reflect  some  of  the 
subjective  opinions  of  the  analyst  concerning  changes  in  the 
variables  used  to  derive  the  cost  estimate.  For  example,  the 
analyst  has  to  choose  an  adjustment  factor  for  an  analogous 
estimate.  He  must  also  make  a  subjective  choice  of  the 
present  and  future  inflation  during  the  life  of  the  program. 
The  use  of  sensitivity  analysis  allows  the  cost  analyst  and 
manager  to  estimate  the  interval  over  which  the  estimate 
varies  when  the  independent  variables  (used  to  derive  the 
estimate)  are  varied  over  their  possible  ranges.  While  sensi¬ 
tivity  analysis  does  not  make  use  of  probability  theory,  it 
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does  provide  a  means  for  the  manager  to  select  the  most 
likely  value  of  the  future  cost  of  the  new  weapon  system 
from  the  interval  derived. 

Some  of  the  cost  estimating  techniques  use  statisti¬ 
cal  manipulation  of  historical  data  to  come  up  with  an 
objective  measure  of  the  total  cost  of  a  system.  Risk  tech¬ 
niques  have  also  been  applied  to  derive  an  objective  measure 
of  total  system  cost.  By  analyzing  historical  data,  a  prob¬ 
ability  distribution  function  was  derived  which  represented 
not  only  the  possible  ranges  of  cost,  but  also  the  most 
likely  value.  This  objective  measure  was  found  to  apply 
best  for  production  situations  where  adequate  historical 
data  is  available.  In  R&D  situations  though,  there  was  not 
a  good  historical ' basis  to  determine  an  objective  estimate 
of  the  probability  density  function.  Yet,  a  probability 
density  function  Is  needed  in  order  to  apply  the  risk  tech¬ 
niques  described  In  this  chapter.  Sometimes  the  only  esti¬ 
mate  of  risk  is  based  on  the  manager's  subjective  feelings 
of  risk.  Techniques  have  been  developed  and  are  still  being 
developed  to  quantify  the  manager’s  subjective  probability 
distribution  function.  This  not  only  gives  the  manager  feed¬ 
back  concerning  his  preferences  (or  bias)  for  taking  risks, 
but  also  allows  him  to  better  evaluate  the  risk  involved 
for  each  of  the  alternatives  he  has  to  evaluate. 

The  initial  cost  estimate  of  a  new  system  is  based 
on  an  analysis  of  the  historical  data  of  a  different  but 
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similar  system  which  is  already  in  existence.  The  data 
base  must  be  adjusted  for  differences-  in:  the  categories 
used  to  classify  costs,  the  physical  and  performance  char¬ 
acteristics,  price  changes,  the  quantity  bought,  type  of 
contract,  manufacturing  techniques,  etc.  (29:17-32).  With¬ 
out  a  sound  data  base  and  a  knowledge  of  its  reliability, 
the  cost  estimate  including  risk  assessment  can  have  built- 
in,  unknown  errors. 

There  are  three  prerequisites  for  cost  estimating 
research  and  analysis  to  have  a  meaningful  impact  on 
the  decision  making  process  in  any  public  agency  or 
private  enterprise.  They  are: 

Policy 

Methodology  and  tools 

Credible  data 

These  prerequisites  form  a  triad  for  effective  deci¬ 
sion  making.  ...  If  either  of  the  three  are  missing 
or  weak,  then  we  are  bound  to  draw  faulty  conclusions 
and  make  less  than  optimal  decisions  [7:1]. 

Although  there  are  many  techniques  to  model  risk  and  uncer¬ 
tainty,  it  is  still  difficult  to  derive  a  subjective  prob¬ 
ability  distribution,  especially  in  the  early  stages  of  an 
R&D  project.  There  is  still  a  built-in  error  in  all  tech¬ 
niques  since  the  input  distribution  in  the  early  stages 
cannot  necessarily  be  modeled  accurately. 

Interview  Results 

As  stated  earlier  in  Chapter  2  under  Interviews, 
the  subject  of  setting  aside  money  to  cover  risk  and  uncer¬ 
tainty  is  politically  a  very  sensitive  issue.  The 


following  paragraphs  contain  a  summary  of  interviews  con¬ 
ducted  at  various  levels  of  DOD,  including  Army,  Navy,  and 
Air  Force. 

Any  good  program  manager  realizes  that  the  early 
phases  of  research  and  development  are  characterized  by  high 
risk,  since  the  concepts  are  still  being  defined,  and  many 
times  there  has  been  no  forerunner  on  which  to  base  an 
accurate  cost  estimate.  While  all  three  services  admit  to 
looking  at  risk  analysis  and  uncertainty,  they  remain  some¬ 
what  silent  on  the  use  of  management  reserves.  This  was 
consistent  with  the  literature  review. 

The  program  manager  is  faced  with  unknown  changes 
in  his  program  brought  about  by  technological  advances, 
changing  system  requirements,  changing  mission  requirements, 
unexpected  cuts  by  Congress  (i.e.,  budget  cuts,  program  can¬ 
cellations),  etc.  In  order  to  cope  with  these  situations 
the  program  manager  has  to  build  extra  money  into  his  pro¬ 
gram.  These  monies  are  termed  "management  reserves"  unof¬ 
ficially  and  represent  an  informal  way  of  funding  for  risks 
and  uncertainty.  Officially,  there  is  no  "management 
reserve";  rather,  the  Engineering  Change  Order  (ECO)  Line 
contains  the  extra  money  needed  to  cover  unplanned  expendi¬ 
tures.  There  also  seems  to  be  some  confusion  concerning  the 
meaning  of  management  reserve  and  risk  and  uncertainty  mon¬ 
ies.  Some  managers  consider  the  terms  one  and  the  same. 

The  Army  is  quite  explicit  that  management  reserves  are  not 
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the  same  as  risk  and  uncertainty  monies.  Essentially,  risk 
mone-y  is  allocated  based  upon  a  formal  analysis  of  uncer¬ 
tainty,  but  some  managers  feel  that  management  reserve  is  a 
"guess"  or  "fudge"  factor  without  any  quantitative  basis. 

Of  all  the  services,  the  Army  appears  to  be  the  only 
service  that  uses  a  codified  and  Congressionally  approved 
method  of  taking  into  account  the  risk  and  known  uncertain¬ 
ties  of  a  project  with  the  associated  costs.  One  interviewee 
stated  that  the  Navy  had  no  single  codified  policy  but  that 
the  program  manager  had  undisclosed  ways  of  providing  the 
necessary  slack  to  cover  risk  and  uncertainties.  It  was 
stated  that  there  is  no  codified  system  in  the  Air  Force  for 
management  reserve.  Risk  is  mentioned  in  AFR  173-1,  but  not 
management  reserve. 

It  was  asserted  that  the  Air  Force  had  looked  at  the 
Army's  concept  of  TRACE  and  provided  a  response  to  Carlucci's 
initiative  pertaining  to  Budget  Funds  for  Technological  Risk 
(Number  Eleven).  In  essence,  the  Air  Force  evaluation  indi¬ 
cated  that  it  was  already  using  methods  which  were  essen¬ 
tially  equivalent  to  the  Army's  TRACE  concept  to  quantify  and 
budget  for  risk,  and  that  the  Air  Force  distributes  all  funds 
to  the  program  manager  except  in  unique  circumstances.  It 
was  said  that  the  main  difference  between  the  Army's  method 
of  handling  risk  monies  and  the  Air  Force's  is  centralized 
management  versus  decentralized  management.  It  also  was 
stated  that  the  concept  of  visible  management  versus 
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hidden  management  reserves  constituted  the  "horns  of  a 
dilemma."  In  other  words,  if  the  money  is  visible,  then  it 
could  be  taken  away,  thus  crippling  a  program.  On  the 
other  hand,  a  program  that  encourages  hidden  reserves  has  no 
means  to  evaluate  the  manager's  performance  in  controlling 
risks.  Chapter  4  will  present  a  detailed  description  of  the 
Air  Force  risk  model  and  the  Army's  TRACE  method. 

One  staff  member  stated,  "Current  Air  Force  practices 
make  management  reserve  a  smart  technique."  In  some  cases 
the  Air  Force  does  use  sophisticated  techniques  for  risk 
analysis.  Management  reserve  is  employed,  but  mostly  by 
"rule  of  thumb"  based  on  historical  data.  It  Is  very  diffi¬ 
cult  to  quantify  risk,  and  certain  programs  are  more  risky 
than  others.  According  to  a  budgeting  specialist,  most  of 
the  risk  money  Is  contained  In  the  ECO  (Engineering  Change 
Order)  line  of  a  budget.  He  also  stated  that,  after  a  pro¬ 
gram  is  completed,  the  amount  of  ECO  and  management  reserve 
used  in  a  program  is  not  readily  apparent  when  reviewing 
the  historical  data.  Another  comment  was  that  management 
reserve  is  only  a  problem  because  there  are  too  many  efforts 
trying  to  make  it  visible. 

The  main  issue  in  dealing  with  risk  money,  uncer¬ 
tainty  and/or  management  reserve  centers  on  the  Carlucci 
initiative  (Budget  Funds  for  Technological  Risk,  Number 
Eleven)  which  requires  management  reserves  to  be  visible 
(up  front).  This  is  perceived  as  a  problem  by  some,  as 
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there  are  expressed  fears  concerning  the  results  of  an  up 
front  management  reserve  system  versus  hidden  management 
reserves.  Past  reprogramming  efforts  have  resulted  in  the 
risk  money /management  reserves  being  removed  from  a  program 
at  times.  It  appears  to  be  generally  accepted  that  if  all 
risk  money/management  reserves  were  openly  identified  for 
all  programs,  any  Congressional  budget  cut  would  remove  the 
risk  money  from  all  the  programs,  which  could  cripple  their 
chances  of  success.  Under  the  present  hidden  system,  the 
Air  Force  is  allowed  to  select  those  programs  to  cut  which 
it  considers  to  be  of  low  priority. 

Program  managers  are  very  success-oriented.  Because 
of  this,  and  because  the  management  of  success  of  present 
day  systems  is  based  upon  program  completion  within  the 
budgeted  cost,  many  alternatives  are  not  considered.  Accord 
ing  to  one  interviewee,  there  is  no  real  interchange  between 
the  cost  people  and  technical  people,  which  results  in  singl 
point  estimates  being  used. 

In  summary.  Lieutenant  Colonel  John  D.  Edgar,  USAF, 
Director  of  the  Research  Directorate  in  Defense  Systems  Man¬ 
agement  College's  Department  of  Research  and  Information, 
wrote  in  an  article 

Existing  Department  of  Defense  (DOD)  and  service 
policy  directives  clearly  state  that  program  managers 
should  indicate  bands  of  uncertainty  in  cost  estimates 
and  "include  risk  costs  and  costs  of  likely  contingen¬ 
cies.  ..."  In  the  past,  however,  program  managers 
who  explicitly  requested  funds  to  cover  program  uncer¬ 
tainties  have  usually  found  those  funds  deleted  by  the 
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services  or  DOD  in  the  planning,  programming,  and  bud¬ 
geting  system  (FPBS),  by  the  Office  of  Management  and 
Budget  (OMB),  or  by  the  Congress.  Thus,  when  such 
adverse  events  actually  occurred,  the  program  manager 
was  faced  with  some  hard  decisions.  Ke  could  delay 
his  program  while  trying  to  obtain  additional  funds 
through  the  PPBS  or  by  reprogramming;  or  he  could 
adjust  funds  internally  by  reducing  the  program  scope, 
stretching  schedules,  deleting  redundancy  in  tasks  and 
hardware,  and  borrowing  against  the  future  by  deleting 
test  hardware,  and  reliability  and  maintainability  tasks. 
Most  program  managers,  whether  they  would  publicly  admit 
it  or  not,  have  responded  to  this  problem  by  budgeting 
an  undisclosed,  internal  management  reserve.  This 
reserve  might  be  spread  in  small  amounts  across  all  the 
program  tasks,  or  concentrated  in  the  form  of  a  single, 
"soft"  work  element  that  could  be  deleted  without  affect¬ 
ing  the  program  [13:60]. 


CHAPTER  4 


TRACE/AIR  FORCE  RISK  MODEL — BRIEF  DESCRIPTIONS 

Overview 

Interview  data  Indicated  that  two  models  would  sat¬ 
isfy  the  requirements  of  the  Carlucci  initiative  "Budget 
Funds  for  Technological  Risk" — the  Army's  TRACE  (Total  Risk 
Assessing  Cost  Estimate)  Model  and  the  Air  Force  RISK  Model. 
Before  discussing  these  models,  the  Monte  Carlo  Technique 
used  in  both  will  be  briefly  described.  This  technique  is 
used  to  derive  a  probability  distribution  which  represents 
the  future  total  cost  of  a  system,  including  an  allowance 
for  risk  and  uncertainties.  Following  the  discussion  of 
TRACE  and  the  Air  Force  RISK  Model,  a  comparison  of  both 
models  will  be  presented. 

Monte  Carlo  Technique 

The  Monte  Carlo  Technique  is  used  to  generate  a 
sample  distribution  from  limited  sized  input  data  which  may 
have  an  unknown  probability  distribution  (contains  uncer¬ 
tainty).  The  input  can  be  described  by  a  probability  func¬ 
tion  selected  by  the  analyst  to  represent  either  a  statisti¬ 
cal  or  a  subjective  representation  of  the  uncertainty  con¬ 
tained  in  the  input  data  (26:7).  The  beta  distribution  is 
most  often  used,  as  is  mentioned  in  Chapter  3  of  this  report,- 
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to  obtain  a  risk  value  within  a  given  confidence  level  to 
represent  the  uncertainty  contained  in  the  input  data.  The 
technique  is  generally  used  as  a  computer  simulation  model 
when  large  amounts  of  input  variables  must  be  processed  in 
order  to  obtain  a  potential  distribution  of  the  total 
future  cost  of  a  system.  For  small  amounts  of  data,  a  hand 
calculator  and  random  number  tables  can  be  used  to  calculate 
the  total  future  cost  of  a  system. 

The  method  generally  functions  by  taking  a  random 
single  valued  sample  from  each  of  the  input  factors  and  com¬ 
bining  them  according  to  the  requirements  of  the  system  cost 
analysis  model  to  produce  an  output  that  represents  the  sys¬ 
tem  cost  uncertainty.  Each  of  the  input  factors  can  be 
represented  by  the  beta  distribution  function  by  specifying 
the  most  likely  value,  the  maximum  value,  and  the  minimum 
value.  When  all  these  variables  are  thus  represented,  the 
computer  then  derives  a  cumulative  distribution  function  for 
each  which  can  then  be  plotted.  This  function  represents  the 
actual  or  estimated  uncertainty  contained  in  the  input  data 
(26:8).  The  following  paragraph  describes  how  this  sample 
is  derived  from  the  input  data. 

In  order  to  keep  the  explanation  simple,  only  one 
variable  "x"  will  be  used,  with  "y"  representing  the  prob¬ 
ability  (see  Figure  6). 

From  the  probability  density,  Y  =  f(x),  describing 
the  actual  (or  estimated)  input  uncertainty,  a  cumula¬ 
tive  distribution  is  plotted.  Next,  a  random  decimal 
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between  zero  and  one  is  selected  from  a  table  of  random 
digits.  By  projecting  horizontally  from  the  point  on 
the  Y-axis  corresponding  to  the  random  decimal  to  the 
cumulative  curve,  we  find  the  value  of  x  corresponding 
to  the  point  of  intersection.  This  value  is  taken  as 
a  sample  value  of  x. 

The  result,  if  this  procedure  is  repeated  numerous 
times,  is  a  sample  of  input  values  that  approximates 
the  required  input  uncertainty.  As  seen  in  Fig.  5 
[see  Figure  7],  the  more  repetitions,  the  better  the 
simulated  input  distribution  [26:8], 

In  order  to  fully  understand  how  the  output  which 
represents  the  total  cost  including  uncertainty  is  derived, 
a  simple  example  is  now  presented.  Suppose  the  total  train¬ 
ing  cost  is  needed  in  order  to  fund  a  training  program.  The 
analyst  found  that  the  total  training  cost  in  dollars  is 
equal  to  the  number  of  personnel  required  times  the  unit 
training  costs  (dollars  per  man) (26:9).  The  analyst  would 
represent  the  amount  of  uncertainty  in  each  of  the  variables 
that  produce  the  total  cost  with  a  beta  distribution  func¬ 
tion  which  he  would  derive  for  each.  These  functions  would 
represent  the  uncertainty  involved  in  his  estimate  of  each 
of  the  cost  factors.  The  beta  distributions  could  be  dif¬ 
ferent  since  the  analyst  might  find  that  he  was  more  certain 
of  the  training  costs  than  of  the  manpower  requirements. 

He  would  input  these  distributions  into  the  computer  along 
with  his  cost  model.  The  computer  would  use  these  inputs 
to  derive  the  cumulative  distribution  function  for  each 
input  variable. 

Using  a  random  number  generator,  the  computer  would 
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enter  the  cumulative  distribution  function  for  each  variable 
and  select  the  corresponding  sample  value  for  both  the  train¬ 
ing  cost  and  the  manpower  requirements.  These  two  values 
are  multiplied  together  to  obtain  a  total  training  cost. 

The  computer  repeats  this  procedure,  using  a  different  ran¬ 
dom  number  each  time,  for  the  number  of  iterations  selected 
by  the  analyst  (i.e.,  1000  times).  The  output  can  then  be 
represented  as  a  plot  of  the  frequency  distribution  for  the 
total  training  cost.  By  analyzing  this  distribution,  the 
cost  analyst  can  obtain  the  most  likely  cost  (from  the  peak 
value)  and  the  maximum  and  minimum  cost  from  the  endpoints. 

He  can  also  derive  the  probability  from  the  "y"  axis  of  the 
output  plot  of  the  frequency  distribution.  If,  for  example, 
the  analyst  wants  a  total  cost  that  includes  enough  money  to 
cover  a  fifty  percent  risk,  he  would  draw  a  line  parallel  to 
the  "x"  axis  from  the  .5  division  on  the  "y"  axis.  He  would 
then  select  the  point  where  this  line  crossed  the  distribu¬ 
tion  curve  and  read  the  value  directly  below  the  point  on  the 
"x"  axis  (26:9-11). 

In  summary,  this  technique  is  a  method  that  can  be 
used  to  theoretically  simulate  or  derive  the  probability 
density  functions  for  each  input  factor  when  only  a  limited 
number  of  data  points  are  available.  Many  times  the  analyst 
only  knows  a  most  likely  value  and  must  somehow  express  his 
uncertainty  concerning  this  value.  He  can  either  use  statis¬ 
tics,  if  there  is  enough  historical  data  available,  or  he  can 
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utilize  subjective  probabilities  to  express  his  uncertainty 
if  no  historical  data  is  available.  The  beta  function  has 
been  described  as  one  method  by  which  he  may  do  this.  The 
Monte  Carlo  Technique  is  able  to  use  this  input  information 
to  produce  a  sample  population  from  which  randomly  selected 
values  are  obtained  to  estimate  the  total  future  system  cost 
distribution,  as  described  by  the  system  cost  analysis  model 
(26:6). 

Air  Force  RISK  Model 

The  Air  Force  Risk  Model  is  the  name  by  which  the 
Air  Force  Systems  Command  (AFSC)  computer  model  "RISK"  has 
been  called  by  various  Air  Force  personnel  interviewed.  The 
input  work  breakdown  structure  (WBS)  elements  are  described 
by  a  beta  distribution,  simulated  by  a  Monte  Carlo  Technique 
and  an  output  produced  by  the  RISK  model  that  is:  ”1)  a  ’best 
measure  of  central  tendency'  for  the  total  cost  distribution, 
and  2)  a  probability  of  occurrence  for  each  total  cost  inter¬ 
val  comprising  the  total  cost  distribution  [ll:para  2.33-” 

The  median  is  used  to  represent  the  "best  measure  of  central 
tendency"  since  the  mean  "includes  all  the  total  cost  distri¬ 
bution  intervals,  but  could  be  greatly  influenced  by  extreme 
low  or  high  values  if  the  distribution  is  skewed  [11: para  2.3]." 
The  analyst  inputs  variance  for  each  element  of  risk  (see 
Appendix  A) .  The  computer  model  produces  an  output  that  lists’- 
each  interval  evaluated  and  the  probability  of  each  occurring. 
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These  probabilities  are  used  to  produce  a  plot  of  the  cumula¬ 
tive  probabilities  versus  dollars  (see  Appendix  B) . 

For  each  WBS  element  or  category,  the  beta  distribu¬ 
tion  represents: 

1)  [that]  some  costs  within  the  distribution  of  pos¬ 
sible  costs  for  a  given  category  are  more  or  less  likely 
than  other  costs  for  the  same  category,  and  2)  the  shape 
of  the  cost  distribution  is  different  for  different 
categories  [llipara  2.1]. 

The  analyst  fills  out  a  sheet  which  contains  the  following 
information  (see  Appendix  A): 

1)  the  point  estimate  of  the  cost,  2)  the  lowest 
and  highest  costs  possible  for  four  categories  of 
risk,  and  3)  the  variation  of  possible  costs  around 
the  point  estimate  within  this  range  for  each  of  the 
four  risk  categories  [11: para  2.2]. 

The  variance  represents  the  estimator's  assessment 
of  the  certainty  with  which  the  point  estimate  can  be 
known,  irrespective  of  the  category's  cost  extremes. 

A  lower  variance  implies  a  more  peaked  cost  distribu¬ 
tion  clustered  tightly  around  the  point  estimate  and 
high  confidence  in  the  point  estimate.  A  higher  vari¬ 
ance  implies  a  more  rectangular  (flatter)  cost  distri¬ 
bution  and  low  confidence  in  the  point  estimate  [11: 
para  2.2]. 

The  analyst  describes  his  estimate  of  the  variance  as  low, 
medium  low,  medium  high,  and  high  by  use  of  a  numbering  sys¬ 
tem  ranging  from  integer  1,  implying  a  high  confidence,  to 
integer  4,  implying  a  low  confidence  in  the  point  estimate. 
Figure  8  shows  a  pictorial  representation  of  the  above  vari¬ 
ances  . 

The  RISK  Model  output  is  shown  in  Appendix  B.  The 
following  is  a  description  of  the  outputs  of  the  RISK  Model.  - 
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1)  Echo  Check  of  Input  Pile,  .  .  allows  the 
user  to  review  the  file  prior  to  running  the  model. 

2)  Summary  of  Item  Estimates,  Table  3,  provides 
total  cost  estimates  of  category  (WBS)  element  and 
their  percent  of  total  cost. 

3)  Total  Cost  Distribution,  Tables  4a  and  4B  [Table 
5],  describe  interval  probabilities,  cumulative  inter¬ 
val  probabilities,  and  shows  the  difference  between  the 
interval  point  and  the  total  point  estimate. 

4)  Total  Cost  Cumulative  Probability  Distribution, 
Table  5  [Table  6],  graphically  compares  percent  cumula¬ 
tive  probability  versus  different  levels  of  total  cost 
[11: para  2.5]. 

’’The  RISK  Model  uses  the  Interdependent  approach 
because  it  better  represents  the  combination  of  technical 
and  cost  estimating  dependencies  [11: para  2.1].”  Inter¬ 
dependency  Implies  that  a  change  in  one  category  directly 
affects  another  category.  This  is  simulated  by  using  the 
same  random  number  to  generate  an  observation  from  each  WBS 
category.  If  all  the  categories  are  assumed  to  be  indepen¬ 
dent,  a  different  random  number  is  used  to  generate  an  obser¬ 
vation  from  each  category.  For  example,  for  a  two  category 
model,  one  random  number  would  be  generated  and  used  to 
obtain  an  observation  from  the  first  category.  The  computer 
would  then  generate  a  new  random  number  and  use  it  to  obtain 
the  second  observation  from  the  second  category.  These  two 
observations  would  then  be  combined  according  to  the  cost 
equation  to  obtain  one  sample  cost  value. 

In  summary,  the  Air  Force  RISK  Model  uses  a  Monte 
Carlo  Technique  to  generate  a  total  cost  distribution  based 
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on  WBS  elements  that  are  input  into  the  model.  These  inputs 
are  represented  by  a  beta  distribution  for  each  element. 

The  output  is  a  Most  Likely  Cost,  the  aggregate  Low  and  High 
Estimates  (Table  5,  Appendix  B),  and  a  Total  Cost  Cumula¬ 
tive  Probability  Distribution  (Table  6,  Appendix  B)  with  the 
probabilities  of  the  various  total  costs  and  the  various 
total  costs  represented  in  graphical  form.  "Management 
reserve  is  the  difference  between  the  greater  total  cost  and 
the  median  [11: para  2.3]"  where  the  median  represents  the  bes 
estimate  of  total  cost.  This  cost  is  "that  total  cost  value 
whose  cumulative  probability  of  occurrence  is  50 %  [ll:para 
2.3]*"  The  RISK  Model  has  the  potential  ability  to  allocate 
the  management  reserve  (or  risk  money)  for  each  fiscal  year. 
The  model  is  also  capable  of  determining  the  quarterly  status 
(or  semiannual)  of  the  "management  reserve  in  relationship  to 
the  previous  estimate  [ll:para  2,7]." 

Army's  TRACE  Method 

The  Total  Risk  Assessing  Cost  Estimate  (TRACE)  is 
composed  of  two  parts:  1)  "Operational  Flow"  or  the  mechan¬ 
ics  of  managing  the  TRACE  deferral  funds  and  2)  the  TRACE 
Methodology  (10:2-3) (see  Figure  9).  Before  discussing  each 
part,  however,  the  TRACE  terminology  will  be  discussed. 

After  this  discussion,  the  operational  flow  will  be  reviewed 
followed  by  a  description  of  the  TRACE  Methodology. 

The  mechanics  of  managing  the  TRACE  program  was 
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TRACE  (Total  Risk  Assessing  Cost  Estimate) — Refers  to  the 
Total  RDT&E  project  cost  which  is  the  sum  of  the  Baseline 
Cost  Estimate  and  the  Risk  Cost. 

BASELINE  COST  ESTIMATE  (BCE)— The  project  cost,  sometimes 
referred  to  as  the  engineering  cost  estimate,  characterized 
by  the  general  method  of  determining  project  cost  against  a 
fixed  schedule  for  all  planned  activities. 

RISK  COST  (RC) — The  cost  of  project  uncertainties  and  is 
mathematically  the  difference  between  the  TRACE  and  the  BCE. 

RISK  COST/FISCAL  YEAR — The  project  risk  cost  for  a  given  fiscal 
year. 

TRACE  DEFERRAL — The  money  allocated  and  retained  by  DA  (Depart¬ 
ment  of  the  Army)  for  project  uncertainties. 

BUDGET — That  money  given  to  projects  to  cover  planned  activi¬ 
ties. 


(Author's  Note:  The  definition  provided  above  Implies  that 
risk  cost  is  driven  by  the  sum  of  the  derived  probability 
distributions  for  cost  element  uncertainties.  Obviously 
there  will  be  some  unknown  uncertainties  which  have  not  beer, 
included .  ) 

Figure  9.  Total  Risk  Assessing  Cost  Estimate 
(TRACE)  Guide  [10:4]. 
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referred  to  as  the  "Operational  Plow"  in  the  TRACE  Guide 
(10).  The  process  begins  when  the  program  manager  determines 
the  initial  recommended  TRACE  values  which  includes  a  pro¬ 
posed  allocation  of  the  risk  money  by  fiscal  year  over  the 
life  of  the  program.  The  TRACE  estimate  is  then  submitted 
through  channels  until  it  is  submitted  to  DOD  and  the  Con¬ 
gress  for  approval  and  funding  of  the  program  described  in 
the  TRACE  estimate.  When  the  appropriation  is  received  by 
HQ  DA,  the  program  manager  receives  the  baseline  estimate 
and  the  TRACE  deferral  funds  are  kept  at  HQ  DA.  The  program 
manager  must  submit  a  request  for  the  TRACE  deferral  funds 
and  must  Justify  their  release  when  he  encounters  a  risk 
situation  (32:17-19).  This  is  essentially  a  centrally  con¬ 
trolled  risk  management  fund.  Notice  that  the  Army  calls 
this  fund  risk  money  and  not  management  reserve.  Some  of 
the  typical  uses  of  TRACE  deferral  funds  that  can  justify 
the  program  manager's  request  are: 

1.  Rescheduling  to  work  around  technical  problems. 

2.  Rescheduling  to  reduce  program  inputs  from 
nominal  budgetary  limitations. 

3.  Design  changes  to  correct  deficiencies  and 
hardware  to  support  the  modifications. 

b .  Additional  testing  to  verify  design  corrections. 

5.  Schedule  slippages  caused  by  late  delivery  of 
components  or  materials. 

6.  Other  actions  for  the  reduction  of  cost  uncer¬ 
tainties  and  cost  impacts  [10:20], 

Hew  did  the  program  manager  decide  what  he  should 
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submit  as  a  TRACE  estimate  for  his  program?  He  applied  one 
of  three  basic  TRACE  methodology  categories:  the  Risk  Factor 
Method,  Probabilistic  Event  Analysis,  and  the  Probabilistic 
Network  Model  (37:5).  A  brief  description  of  each  follows: 

a.  Risk  factor  Method  (RFM).  This  is  the  most 
subjective  of  the  approaches  and  is  based  largely  on 
judgement.  It  consists  of  subjectively  determining 

a  risk  factor  and  multiplying  it  by  a  given  Work  Break¬ 
down  Structure  (WBS)  element. 

b.  Probabilistic  Event  Analysis  (PEA).  This  tech¬ 
nique  is  also  oriented  to  the  WBS  elements  of  a  program 
and  addresses  the  probability  of  a  problem  occurring  for 
each  element,  what  the  problem  will  cost  for  that  ele¬ 
ment,  and  the  probability  of  the  problem  impacting  other 
elements  and  the  associated  costs. 

c.  Probabilistic  Network  Model  (PNM).  The  PNM  is 
basically  a  combination  of  the  Program  Evaluation  and 
Review  Technique  (PERT)  and  Monte  Carlo  simulation. 

The  model  is  activity  oriented,  incorporating  uncer¬ 
tainty  associated  with  cost  and  schedule  for  a  given 
level  of  technical  performance.  A  computer  is  normally 
required  so  as  to  allow  timely  analysis  of  the  program's 
activities  and  their  interactions,  and  assessment  of  the 
technical,  cost,  and  schedule  risks.  Once  accomplished, 
the  PNM  can  serve  as  a  vehicle  to  develop  the  RDT&E 
portion  of  the  BCE,  the  TRACE  deferral,  and  the  TRACE. 

It  also  allows  for  sensitivity  analysis,  cost  tracking, 
and  rapid  reevaluation  of  the  effect  of  program  changes. 
Fiscal  year  costs  can  be  determined  in  the  form  of  cost 
distributions.  Trace  deferrals  can  be  evaluated  for  a 
variety  of  risk  levels.  This  is  the  preferred  method 

of  constructing  the  TRACE  for  complex  programs  [37:6]. 

The  PNM  will  be  discussed  in  more  detail  since  it  is 
the  recommended  method  for  complex  projects.  It  is  not 
exactly  like  the  method  presented  in  the  Air  Force  RISK  Model. 
Appendices  C  and  D  contain  sample  inputs  and  outputs  of  the 
PNM  approach. 

The  TRACE  methodology  (PNM)  uses  a  network  modeling 
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approach  which  is  run  on  a  computer  using  such  computer 
models  as  VERT  or  RISNET  (Risk  Information  System  and  Net¬ 
work  Evaluation  Technique).  Any  valid  model  can  be  used  in 
the  TRACE  methodology.  In  selecting  a  model,  "the  major 
consideration  is  that  the  model  should  sufficiently  demon¬ 
strate  the  management  philosophy  and  interrelationships 
between  activities  [10:25]."  Once  this  model  has  been  chosen, 
the  cost  and  schedule  data  associated  with  the  network  activi¬ 
ties  are  input  into  the  computer  with  a  description  of  the 
analysts  choice  of  probability  distributions  to  represent 
the  uncertainty  in  each  element  of  data  (10:26). 

In  most  cases  an  analyst,  using  his  experience,  and 
the  experience  of  this  data  source,  can  describe  uncer¬ 
tainty  in  terms  of  a  uniform  distribution,  a  symmetrical 
triangular  distribution,  or  a  skewed  triangular  distri¬ 
bution.  Other  distributions  are  available  and  may  be 
used  when  the  conditions  are  appropriate  [10:26]. 

Appendix  C  contains  an  example  of  inputs  using  the  networking 

methodology. 

RISNET  is  a  computer  program  package  that  was  not 
described  in  any  great  detail  In  any  of  the  TRACE  documents. 
RISNET  was  described  as  a  computer  program  package  that  sup¬ 
ported  the  TRACE  network  model  technique  (TRACENET).  "TRACE- 
NET  is  a  Monte  Carlo  network  simulation  program  which  uses 
the  same  network  logic  as  RISCA  III  and  incorporates  [37:17]" 
an  Iterative  technique  which  accomplishes  "the  yearly  alloca¬ 
tion  of  TRACE  deferral  while  still  making  allowances  for  the 
PMfs  preferences  or  aversion  to  risk  [37:1*0." 
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Appendix  D  gives  an  example  of  the  outputs  of  the 
computerized  TRACE  network  methodology.  The  necessary  sta¬ 
tistical  outputs  which  are  needed  to  perform  a  TRACE  analy¬ 
sis  are: 


1.  Total  cost  distribution 

2.  Total  cumulative  cost  distributions 

3.  Cost  distributions  for  each  fiscal  year 

4.  Allocations  of  risk  cost  per  fiscal  year  [10:26]. 
These  computer  produced  outputs  are  used  by  the  analyst  to 
prepare  the  information  and  make  a  chart  of  the  desired 
information  as  shown  in  Figure  10.  The  TRACE  "is  simply  the 
point  on  the  cost  distribution  corresponding  to  a  desired 
level  of  risk  taking.  The  recommended  TRACE  should  not  be 
less  than  the  50/50  point  .  .  .  [10:26]."  The  point  can 

be  chosen  based  on  the  manager's  judgement  and  whatever 
trade-offs  he  feels  are  necessary.  TRACE,  Budget ,  and  Risk 
Cost  are  defined  earlier  (see  Figure  9).  Expected  Loss  is 
defined  as  that  expected  cost  which  would  result  from  any 
given  fiscal  year,  should  there  be  an  overrun  in  that  year 
(10:29). 

In  Figure  10,  the  expected  loss  is  in  reality  an 
expected  value  calculation  but  it  is  not  the  same  as  the 
TRACE  Level.  A  TRACE  Level  of  50$  is  defined  as  that  point 
where  there  is  "a  50/50  chance  of  producing  either  a  cost 
overrun  or  an  underrun  [36:3].”  A  choice  of  70$  would  sug¬ 
gest  a  70$  chance  of  success  and  a  30$  chance  of  failure  (a 
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Figure  10.  Risk  Cost  Allocation  for  TRACE  Risk  Levels  [11:43] 


cost  overrun).  In  other  words,  the  actual  costs  are  between 
the  extreme  of  "funding  for  all  possible  risks  and  uncertain¬ 
ties,  [and  the  other  extreme  of]  funding  for  those  activities 
which  can  be  identified,  costed,  and  scheduled  with  certainty 
[37:2]."  The  recommended  TRACE  Level  is  50$  but  the  analyst 
can  choose  a  level  which  represents  the  risk  that  the  actual 
cost  will  be  greater  than  the  estimate  which  is  acceptable 
to  him  (37:2).  Obviously,  there  is  some  subjective  judgement 
required  in  selecting  a  TRACE  level. 

Comparison  of  TRACE  and  Air  Force  RISK  Models 

Both  models  use  a  Monte  Carlo  simulation  technique. 
They  both  generate  an  output  that  represents  the  most 
likely  cost  of  the  program.  They  each  are  also  able  to  . 
present  in  graphical  form  the  cumulative  probability  distri¬ 
bution  of  the  total  costs.  They  also  are  similar  In  that 
the  analyst  is  required  to  input  the  data  in  some  form  which 
represents  the  uncertainty  found  in  each  data  element  or 
category. 

Although  they  have  similarities,  there  are  some  dif¬ 
ferences.  The  TRACE  methodology  not  only  produces  a  total 
cumulative  cost  distribution,  but  is  able  to  produce  prob¬ 
ability  distributions  for  each  fiscal  year  over  the  life  of 
the  project.  It  is  also  able  to  predict  an  expected  loss 
value  which  can  be  used  to  determine  if  the  amount  of  risk 
money  projected  for  any  fiscal  year  is  adequate. 
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The  Air  Force  RISK  Model  places  more  emphasis  on 
the  use  of  WBS  elements  and  the  description  of  their 
variances  and  uncertainty  distributions,  while  the  Army's 
TRACE  Model  allows  a  choice  of  either  a  network  model, 
probabilistic  event  analysis  which  addresses  WBS  elements 
specifically,  or  the  use  of  a  risk  factor  method  which  relies 
mostly  or.  the  analyst's  subjective  judgement.  This  does  not 
appear  to  be  a  major  problem,  since  the  Air  Force  method 
allows  the  analyst  to  use  either  objective  or  subjective 
methods  in  describing  the  input  data.  The  Air  Force  model 
takes  into  account  the  interdependencies  that  are  repre¬ 
sented  in  networking  moiels  by  requiring  the  Monte  Carlo 
technique  to  apply  the  same  random  number  to  each  input 
element  distribution  when  calculating  the  total  cost  in  any 
iteration.  The  Air  Force  Model  can  also  be  run  quarterly 
to  determine  the  management  reserve  or  risk  money  status,  as 
well  as  to  allocate  this  money  for  each  fiscal  year. 

Although  their  example  did  not  show  this  (see  Appendices 
A  and  B),  Paragraph  2.7  of  their  document  mentioned  this 
ability  (11).  The  Air  Force  mentioned  that  there  are  future 
enhancements  possible  for  their  model  (11: para  2.6). 

The  major  area  of  contention  in  comparing  the  two 
models  is  based  on  the  management  philosophy  advocated  by 
the  TRACE  method.  The  Air  Force  prefers  a  decentralized 
style  of  management,  while  the  Army  prefers  a  centrally  con¬ 
trolled  style.  It  is  the  author's  opinion  that  the  TRACE 
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methodology  would  be  more  acceptable  to  the  Air  Force  If 
this  requirement  for  a  centrally  managed  fund  would  not  be 
required.  The  Air  Force's  RISK  Model  certainly  does  recog¬ 
nize  the  need  for  risk  money3  which  they  term  management 
reserve .  The  Army  avoids  the  term  management  reserve,  since 
it  represents  to  those  Army  officla. •  . nterviewed  a  source  of 
money  set  aside  to  cover  unplanned-for  events  without  any 
attempt  to  analyze  the  risk  associated  with  the  uncertainty 
of  the  future.  In  other  words }  it  can  be  considered  a  "slush 
fund"  with  no  legitimate  purpose. 

Both  models  have  the  following  advantages: 

1.  The  cost  uncertainty  associated  with  the  inter¬ 
actions  of  the  input  variables  are  considered. 

2.  Subjective  as  well  as  objective  prcbabilities 
are  considered.  This  includes  an  incorporation  of  the 
project  manager's  risk  preference. 

3.  Both  models  provide  an  output  which  represents 
the  cumulative  probability  of  the  total  system  cost. 

4.  Both  methods  use  a  Monte  Carlo  Sampling  Tech¬ 
nique  which  allows  the  theoretical  derivation  of  a  sample 
population's  parameters  when  that  data  would  not  be  available 
in  the  data  base. 

Both  Models  have  the  following  disadvantages: 

1.  Both  require  a  high  level  of  analytical  skill 
on  the  part  of  the  analyst. 
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CHAPTER  5 


CONCLUSIONS 


Overview 

This  chapter  presents  a  summary  of  the  results  of 
the  research  effort.  As  stated  earlier  in  Chapter  1,  this 
effort  was  limited  to  the  use  of  management  funds  to  cover 
the  extra  costs  due  to  uncertainty.  This  chapter  includes 
a  discussion  of  the  conclusions  that  were  based  on  the 
research . 

Summary 

It  was  found  that  three  areas  need  to  be  considered 
in  the  management  of  risk  and  uncertainty:  methodology  and 
techniques,  policy,  and  data  base.  In  reviewing  management 
techniques  in  Chapter  3,  it  was  found  that  managers  must  be 
aware  of  the  various  factors  that  contribute  to  cost  growth. 
They  must  also  understand  how  these  factors  contribute  to 
the  uncertainty  involved  in  a  project.  In  order  to  under¬ 
stand  the  meaning  of  risk  and  uncertainty,  they  should  have 
some  understanding  of  probability  theory  and  statistical 
methods.  Risk  is  nothing  more  than  the  result  of  analyzing 
and  quantifying  the  uncertainty  involved  in  any  endeavor. 
Various  techniques  have  been  proposed  in  order  to  better 
manage  R&D  programs  within  government.  It  is  apparent  from 


the  literature  review  that  there  is  no  one  universal  tech¬ 
nique  to  completely  manage  every  program.  Since  each  project 
and  each  organization  developed  to  manage  a  project  are 
somewhat  unique,  the  risk  and  uncertainty  management  policies 
for  that  program  must  also  have  some  program-specific  fea¬ 
tures  . 

The  techniques  used  to  estimate  costs  were  reviewed 
in  order  to  understand  their  shortcomings  in  identifying 
risk.  It  was  found  that  they  fell  short  of  quantifying 
uncertainty  into  a  calculable  risk.  Therefore,  further 
research  was  conducted  concerning  the  various  methods  of  risk 
analysis.  It  was  found  that  objective  techniques  to  statis¬ 
tically  derive  the  probabilities  of  risk  based  on  a  prior 
similar  system  were  not  always  possible,  especially  in  R&D. 
The  analyst's  subjective  observations  and  experience  were, 
many  times,  the  only  evaluations  that  could  be  made  concern¬ 
ing  past  historical  data  as  applied  to  a  new  system.  It  was 
also  found  that  various  managers  and  analysts  have  different 
propensities  for  risk  taking  or,  more  commonly,  have  a  point 
of  risk  aversion.  Various  techniques  are  now  in  development 
to  aid  the  manager  in  understanding  his  risk  aversion. 

These  techniques  are  designed  to  reveal  at  what  point  any 
given  manager  or  analyst  will  not  take  a  risk,  but  will 
require  either  extra  information  or  will  include  a  certain 
amount  of  extra  money  (management  reserve)  to  cover  unex¬ 
pected  and  unquantifiable  risks. 
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It  was  found  that  there  is  no  universal  DOD  policy 


at  this  time.  The  Carlucci  initiatives,  Budget  to  Most 
Likely  Cost  and  Budget  Funds  for  Technological  Risk,  are 
presently  being  evaluated  by  each  service.  Of  the  three 
services,  the  Air  Force  and  the  Army  have  proposed  methods 
in  response  to  the  Carlucci  initiatives.  This  does  not 
imply,  though,  that  the  services  have  not  been  considering 
risk  in  their  projects.  As  the  literature  review  has  shown, 
risk  and  uncertainty  has  been  considered  by  all  the  services 
since  the  late  1960s. 

The  Army's  TRACE  and  the  Air  Force's  RISK  Models 
were  discussed  in  Chapter  4.  It  was  found  that  they  were 
both  capable  of  projecting  the  future  total  cost  of  a  s:  . 
based  upon  cost  elements  that  were  selected  by  the  analyst. 
While  TRACE  provided  more  graphical  outputs  and  information 
than  the  Air  Force  RISK  Model ,  the  RISK  Model  could  be  modi¬ 
fied  to  do  the  same  in  the  future.  At  the  moment,  the  main 
issue  separating  the  two  is  the  requirement  of  a  centrally 
controlled  risk  fund.  The  Air  Force  prefers  a  decentralized 
management  style,  while  the  Army  uses  a  centrally  managed 
system.  Although  the  Air  Force  does  not  find  TRACE  accept¬ 
able,  it  might  consider  the  incorporation  of  some  of  the 
TRACE  features  into  its  own  RISK  Model. 

None  of  these  models  operate  well  without  a  good 
data  base.  The  data  base  must  be  constantly  updated  to 
reflect  new  information  as  It  becomes  available.  Any  old 


and  outdated  information  should  be  removed  and  stored.  It 
should  also  be  remembered  that  not  every  new  system  can  be 
accurately  estimated  from  the  data  base  information,  espe¬ 
cially  when  state-of-the-art  advances  are  incorporated  into 
the  new  system. 

The  issue  concerning  the  use  of  management  reserves 
for  risk  and  uncertainty  is,  at  the  moment,  a  political 
issue.  Carlucci's  initiatives  are  an  attempt  to  formally 
recognize  the  need  for  a  management  reserve  and/or  risk 
money  within  DOD  projects.  As  more  and  more  Congressional 
leaders  and  upper  level  management  gain  confidence  that  this 
management  reserve  is  necessary  and  will  not  be  misused,  .he 
political  fervor  will  recede  and  me  -e  managers  will  be 
willing  to  discuss  the  matter  oper.l  .  The  use  of  mar’  cement 
reserve  in  any  project  is  Just  good  business  sense. 

No  one  approach  appears  to  be  clearly  superior  tc 
any  other.  They  each  have  their  merits.  As  new  methodolo¬ 
gies  come  into  practice  and  better  management  techniques 
and  policies  are  developed,  DOD's  ability  to  budget,  manage, 
and  estimate  will  continue  to  improve,  as  it  has  since  the 
1960s. 

The  research  objectives  were  met.  Management  reserv 
practices  have  been  reviewed,  cost  estimating  and  risk  man¬ 
agement  techniques  have  been  identified  and  evaluated,  the 
leading  issues  concerning  the  use  of  management  reserves  in 
risk  and  uncertainty  management  have  been  described,  and 
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the  leading  techniques  have  been  identified  and  an  illustra¬ 
tion  of  their  use  has  been  provided  in  the  Appendices.  This 
^summary  has  described  the  above  research  objectives.  The 
conclusions  which  follow  address  the  research  questions  that 
are  found  in  Chapter  1  of  this  report. 

Conclusions 

The  underlying  problem  found  in  this  report  is  not 
whether  management  reserve  should  be  applied  in  a  govern¬ 
ment  project,  but  rather  what  kind  of  confidence  does  Con¬ 
gress  and  other  upper  level  management  have  in  the  DOD  meth- 
o'-  ogy  of  program  management.  The  present  DOD  program  man¬ 
agement  methodology  manages  the  risk  and  uncertainty  as  well 
as  the  costs  involved  in  a  project  for  which  the  government 
has  appropriated  money  in  order  to  obtain  a  specific  weapon 
system  or  completed  research  project. 

The  Army's  TRACE  concept  has  been  accepted  and  recog 
nized  by  Congress  as  an  effective  means  to  analyze  and  bud¬ 
get  for  risk  in  a  program.  Although  this  technique  fits  the 
Army's  need,  it  should  not  be  inferred  that  all  services 
should  use  this  technique.  After  all,  each  service  is 
organized  in  a  different  manner,  performs  different  missions 
and  has  their  own  style  of  management.  What  should  be 
required  is  that  each  service  analyze  and  provide  some  kind 
of  technique  that  higher  level  management  and  Congress  will 
feel  confident  in  accepting.  This  would  allow  each  service 
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to  develop  a  program  risk  management  technique  based  upon 
its  individual  management  style,  yet  consistent  with  the 
need  for  good  risk  management  in  order  to  remain  within  the 
constraints  of  cost,  schedule,  and  performance. 

Thus,  the  hidden  issue  concerning  management  reserve 
is  not  the  use  of  such  a  reserve  itself,  but  the  perception 
that  Congress  and  higher  level  management  have  of  a  service's 
program  to  accurately  manage  risk  and  uncertainty  without 
exceeding  the  budget  constraints  as  defined  in  the  program. 

In  order  to  achieve  this,  more  objective  statistical  tech¬ 
niques  can  be  used  to  derive  the  baseline  cost  estimate. 
Therefore,  the  present  system  of  submitting  a  poir.c  cost 
estimate  (which  includes  fixed  program  milestones,  fixed 
schedule,  and  fixed  performance  parameters)  must  be  modified 
so  that  additional  information  gained  as  the  program  pro¬ 
gresses  can  be  used  to  get  the  necessary  funds  for  its  com¬ 
pletion.  At  present,  once  a  point  estimate  is  submitted,  it 
becomes  the  controlling  guideline  throughout  the  life  of  the 
program. 

The  present  system  of  measuring  a  program  manager's 
performance  based  on  his  avoidance  of  a  cost  overrun  on  his 
project  forces  the  manager  to  maintain  hidden  reserves  and 
high  estimates  in  order  to  protect  himself  from  the  unknown 
causes  (uncertainties)  that  cause  cost  growth  within  his 
program.  The  manager  should  be  measured  on  the  efficient 
use  of  the  funds  he  controls,  and  whether  he  obtains  the 
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performance  as  specified  when  the  weapon  system  was  con¬ 
ceptualized.  This  has  been  the  basis  of  industrial  reward¬ 
ing  and  control  of  their  managers.  While  profit  can  be 
used  in  industrial  settings  to  measure  a  manager's  perfor¬ 
mance,  the  government,  being  a  non-profit  organization,  can¬ 
not  use  the  same  measure.  Therefore,  the  project  manager's 
ability  to  manage  the  funds  given  him  should. be  a  criteria 
of  his  performance  measurement  and  not  just  the  fact  that  he 
did  not  have  a  cost  overrun. 

While  it  is  important  for  all  the  services  to  have 
a  risk  management  technique  which  the  Congressional  authori¬ 
ties  have  confidence  in,  unless  the  budget  enactment  pro¬ 
cess  deficiencies  are  corrected,  the  DOD  will  not  be  able 
to  make  full  use  of  risk  management  techniques.  For  example, 
it  is  impossible  to  take  advantage  of  quantity  buys  and  other 
cost-saving  techniques  when  program  managers  are  required  to 
obligate  all  their  funds  within  a  year  or  two. of  their  appro¬ 
priation.  It  is  also  difficult  to  obtain  these  cost  savings 
when  a  manager  does  not  even  know  for  certain  whether  his 
program  funding  will  be  cut  from  one  year  to  the  next. 

Until  we  better  resolve  the  internal  management  problems 
which  we  have  built  into  our  government  procurement  system, 
it  will  be  impossible  for  any  risk  assessment  technique  to 
accurately  predict  total  program  costs  when  the  environment 
constantly  changes,  thereby  causing  a  change  in  the  assump¬ 
tions  used  to  estimate  the  cost  of  a  program. 
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Admittedly,  more  and  more  techniques  are  being  devel¬ 
oped  in  order  to  quantify  the  subjective  probabilities 
involved  in  risk  analysis  in  a  new  acquisition  program. 

While  the  development  of  better  objective  techniques  should 
continue,  more  emphasis  should  be  placed  on  the  further 
development  of  subject  probability  estimating  techniques. 
After  all,  at  the  beginning  of  a  new  program,  where  the 
highest  risk  and  uncertainty  occurs,  little  data  is  avail¬ 
able  to  make  an  objective  probability  evaluation.  The  best 
available  method  is  the  use  of  subjective  judgement.  With 
improved  subjective  probability  estimating  techniques,  the 
analyst  will  be  able  to  better  apply  his  subjective  judge¬ 
ment  at  the  start  of  a  program  which  will  result  in  a  more 
accurate  program  baseline  cost.  The  development  of  subjec¬ 
tive  techniques  should  involve  development  of  better  govern¬ 
ment  management  of  the  system  acquisition  process.  Therefore 
at  the  present  time,  there  is  no  one  best  way  to  manage  and 
control  risk  in  any  program.  It  is  senseless  to  hold  the 
manager  totally  responsible  for  overruns,  when  the  majority 
of  the  time  changes  directed  by  higher  level  management. 
Including  Congress,  cause  cost  overruns.  These  are  clearly 
beyond  the  control  of  the  manager. 

In  summary,  there  are  no  generalized  risk  manage¬ 
ment  techniques  that  apply  to  every  situation.  While  it  is 
nice  to  be  able  to  quantify  everything  in  our  environemnt , 
the  reality  is  that  subjective,  qualitative-type  judgements 
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and  estimates  must  still  be  made.  Because  of  the  myriad 
factors  involved  in  real  world  situations,  it  is  impossible 
as  well  as  cumbersome  at  present  to  model  the  entire  pro¬ 
cess  accurately.  In  other  words,  the  universal  risk  model 
which  applies  to  every  situation  and  all  services,  as  well 
as  every  kind  of  system,  does  not  exist.  Therefore,  risk 
minimization  and  control  procedures  must  be  developed  that 
recognize  that  some  cost  overruns  cannot  be  eliminated  but 
only  managed.  Better  risk  management  techniques  are  needed 
rather  than  just  developing  more  mathematical  techniques 
to  quantify  risk. 

Recommendat ions 

1.  Further  research  should  be  done  in  the  manage¬ 
ment  control  of  risk  and  uncertainty,  as  well  as  cost 
schedule  variances  within  government  programs. 

2.  Further  study  should  be  performed  concerning 
managers'  subjective  choices  under  risk  and  uncertainty. 

A  method  should  be  developed  where  an  analysis  could  be 
made  of  the  choices  a  manager  would  make,  given  various  con¬ 
ditions  of  risk  and  uncertainty.  From  this  data,  a  chart 
could  be  prepared  which  would  aid  a  manager  or  even  an 
analyst  in  making  a  selection  of  the  probable  risk  in 
whatever  system  or  sub-system  he  is  evaluating. 
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Risk 


WBS  Element  Name:  _ 

Most  Likely  Estimate: 

Element  of  Risk  Low 

1.  Estimating 

2.  Schedule 

3.  Technology 

4.  Configuration 


Remarks /Rat lonale 


APPENDIX  B 


AIR  FORCE  RISK  MODEL  OUTPUT  (11) 


TABLE  3 

ECHO  CHECK  OF  INPUT  FILE 


System:  Sample 

(Data  In  Millions  FY  1980  dollars) 


WBS  Element  1:  Const 

Most  Likely  Cost 

Estimate:  267.00 

Estimating  Tech/Appl 
Uncertainty 
Program  'Schedule 
Uncertainty 
Technology  Advance 
Uncertainty 
System  Configuration 
Uncertainty 

WBS  Element  2:  GFE 

Most  Likely  Cost 

Estimate:  68.90 

Estimating  Tech/Appl 
Uncertainty 
Program  Schedule 
Uncertainty 
Technology  Advance 
Uncertainty 
System  Configuration 
Uncertainty 

WBS  Element  3:  TIS 

Most  Likely  Cost 

Estimate:  38.90 

Estimating  Tech/Appl 
Uncertainty 
Program  Schedule 
Uncertainty 
Technology  Advance 
Uncertainty 
System  Configuration 
Uncertainty 


Low 

!  . 

Variance 
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275.00 

1 

Low 
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1 
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263.00 

1 

Low 

200.00 
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1 

Low 
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1 
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1 
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68.80 
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1 
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Low 

30.00 
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High 
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0 

0 

m 

3 

Med 

High 

33.20 

60.00 

1 

_L 

Low 

35.00  53.00 


1  Low 


Variance 


TABLE  3 


WBS  Element  4 :  Chng  Order 

Most  Likely  Cost 

Estimate:  40.70 

Estimating  Tech/Appl 
Uncertainty 
Program  Schedule 
Uncertainty 
Technology  Advance 
Uncertainty 
System  Configuration 
Uncertainty 

WBS  Element  5.'  Claims 

Most  Likely  Cost 

Estimate:  13.90 

Estimating  Tech/Appl 
Uncertainty 
Program  Schedule 
Uncertainty 
Technology  Advance 
Uncertainty 
System  Configuration 
Uncertainty 

WBS  Element  6:  Sys  Int 

Most  Likely  Cost 

Estimate:  15.70 

Estimating  Tech/Appl 
Uncertainty 
Program  Schedule 
Uncertainty 
Technology  Advance 
Uncertainty 
System  Configuration 
Uncertainty 


Continued 
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TABLE  5 


TOTAL  COST  DISTRIBUTION 


Sample 

iData  in  millions  FY  1980  dollars) 

(Cost  Uncertainty-Risk  Analysis  with  interdependent  compon 
ents;  date  of  run:  4/16/81) 


Interval 

Range 

Frequency  Prooability 

Cum  Prob 

1 

338.5  - 

342.2 

0 

0.000 

0.000 

2 

342.2  - 

345.5 

0 

0.000 

0.000 

3 

345.9  - 

349.6 

0 

0.000 

0.000 

4 

349.6  - 

353.3 

0 

0.000 

0.000 

5 

353.3  - 

357.0 

0 

0.000 

0.000 

6 

357.0  - 

360.7 

0 
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0.000 

7 

360.7  - 

364.4 

0 
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0.000 

8 

364.4  - 

368. 1 

0 
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0.000 

9 

368.1  - 

371.8 

0 

0.000 

0.000 

10 

371.8  - 

375.5 

0 

0.000 

C.000 

11 

375.5  - 

379  -2 

2 

.001 

.001 

12 

379.2  - 

382.9 

0 

0.000 

.001 

13 

382.9  - 

386.6 

0 

0.000 

.001 

14 

386.6  - 

390.3 

6 

.002 

.003 

15 

390.3  - 

394.0 

3 

.001 

.004 

16 

394.0  - 

397.7 

3 

.001 

.006 

17 

397.7  - 

401.4 

33 

.013 

•  039 

18 

401.4  - 

405  . 1 

31 

.012 

.031 

19 

405.1  - 

408.8 

23 

.009 

.040 

20 

408.8  - 

412.5 

46 

.018 

.059 

21 

412.5  - 

416.2 

75 

.030 

.089 

22 

416.2  - 

419.9 

94 

.038 

.  126 

23 

418.9  - 

423.6 

61 

.024 

.  151 

24 

423.6  - 

427.3 

107 

.043 

.  194 

25 

427.3  - 

431.0 

97 

.039 

.232 

26 

431.0  - 

434 . 7 

161 

.064 

.297 

27 

434.7  - 

438.4 

123 

.049 

.346 

28 

438.4  - 

442  . 1 

141 

.056 

.  402 

29 

442 .  1  - 

445.8 

146 

.058 

.461 

30 

445.8  - 

449.5 

143 

.059 

.520 

31 

449.5  - 

453.2 

118 

.047 

.567 

32 

453.2  - 

456.9 

137 

.055 

.  622 
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TABLE  5  -  Continued 


Interval 


Range 


Frequency  Probability  Cum  Prob 


456.9 

- 

46 

460.6 

— 

46 

464.3 

— 

46 

468.0 

- 

47 

471.7 

- 

47 

475.4 

- 

47 

479-1 

- 

48 

482.8 

— 

48 

486.5 

- 

49 

490.2 

- 

49 

493-9 

- 

49 

497.6 

- 

50 

501.3 

- 

50 

505.0 

- 

50 

503.7 

- 

51 

512.4 

61 

7 

.679 

9 

.726 

6 

.774 

54 

534.6  -  538.3 

0 

55 

538.3  -  542.0 

0 

0.000 

56 

542.0  -  545.7 

0 

0.000 

57 

545.7  -  549.4 

0 

0.000 

58 

549.4  -  553.1 

0 

0.000 

59 

553.1  -  556.8 

0 

0.000 

60 

556.8  -  560.5 

0 

0.000 

The  aggregate  most  likely  cost  estimate  is 

The  aggregate  low  is 

The  aggregate  high  is 

The  aggregate  mean  is 

The  90  percent  one-sided  confidence 
interval  is 

The  50  percent  one-sided  confidence 
interval  is 


1.000 

1.000 

1.000 

1.000 

1.000 

1.000 

445  .  1 

338.5 

560.5 
449 . 4 

453.6 
448 . 8 


TABLE  6 


TOTAL  COST  CUMULATIVE  PROBABILITY  DISTRIBUTION 


Uncertainty  Distribution  -  Cumulative  Probability 


Percent 

Probability 
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PROJECT  X  TRACE  NETWORK  INPUT  DATA 


PROJECT  X  TRACE  NETWORK  INPUT  DATA 


202  Targets  Procurement  a  =  2000.0;  2000.0;  2300. 


PROJECT  X  TRACE  NETWORK  INPUT  DATA 
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